From daniel.kidger at uk.ibm.com Wed Jun 1 15:55:57 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 1 Jun 2016 14:55:57 +0000 Subject: [gpfsug-discuss] Building portability layer fails In-Reply-To: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk> References: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk>, <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com> Message-ID: <201606011456.u51EuIRY007236@d06av01.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 1 16:03:04 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 1 Jun 2016 15:03:04 +0000 Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All versions In-Reply-To: References: Message-ID: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Wed Jun 1 16:21:28 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Wed, 01 Jun 2016 16:21:28 +0100 Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All versions In-Reply-To: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> References: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> Message-ID: <1464794488.3773.15.camel@buzzard.phy.strath.ac.uk> On Wed, 2016-06-01 at 15:03 +0000, Daniel Kidger wrote: > By the way the files affected are all binaries so not as obvious a > security risk as allowing a backdoor to an interactive ksh shell as > root. Well yeah, Linux has not allowed setuid on scripts of any kind for around 20 years now. Hence even if you are running on AIX they will be binaries because otherwise you would have to maintain a script for AIX and a binary for Linux. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From douglasof at us.ibm.com Wed Jun 1 20:54:04 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Wed, 1 Jun 2016 15:54:04 -0400 Subject: [gpfsug-discuss] IBM Webinar Message-ID: <20160601195415.A2EA26E044@b03ledav001.gho.boulder.ibm.com> Not everyone can make it to a User Group, so we are kicking off a series of webinars with IBM Spectrum Scale experts. Join us for the first one - and bring questions. I will be hosting and having two of our architects on the call. Title: Building a Better Data Ocean: what's new in IBM Spectrum Scale Date: June 16, 12:00 EST Speakers: Piyush Chaudhary, Spectrum Scale Architect, Big Data & Analytics Rob Basham Transparent Cloud tiering and Cloud archive Storage lead architect URL: https://attendee.gotowebinar.com/register/6431264887578253057 -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Wed Jun 1 23:04:45 2016 From: billowen at us.ibm.com (Bill Owen) Date: Wed, 1 Jun 2016 15:04:45 -0700 Subject: [gpfsug-discuss] SECOND CHANCE - OpenStack & Spectrum Scale Usage Survey In-Reply-To: References: Message-ID: <20160601220453.5CD0EC604C@b03ledav006.gho.boulder.ibm.com> If you already filled out the OpenStack & Spectrum Scale Usage Survey, Thank You! If you haven't yet, there's still time to be heard - Please take 10 minutes to answer the questions in this short survey: https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey Thank you, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: Bill Owen/Tucson/IBM To: gpfsug-discuss at gpfsug.org Date: 05/14/2016 05:05 AM Subject: OpenStack & Spectrum Scale Usage Survey We in Spectrum Scale development are very interested in learning about your current & planned usage of OpenStack with Spectrum Scale, as well as plans for deploying containers in your Spectrum Scale environment. We would like to use this information to help shape our road map in this area over the next 18-24 months. Please take 10 minutes to answer the questions in this short survey: https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey Thank you, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Fri Jun 3 14:38:01 2016 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 3 Jun 2016 13:38:01 +0000 Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 on Haswell/Broadwell? Message-ID: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding "nosmap" to the kernel command line in grub prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.) >From https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html, section 3.2: Note: In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor * Disable the Supervisor Mode Access Prevention (smap) kernel parameter * Reboot the RHEL 7 node before using GPFS Some observations worth noting: 1. We've been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs? 2. It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without "nosmap". Instead, an error message describing the fix would have made my day. 3. I'm going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. Thanks, Paul Sanchez -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jun 7 19:25:39 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 7 Jun 2016 11:25:39 -0700 Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 onHaswell/Broadwell? In-Reply-To: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> References: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> Message-ID: Hi Paul, Yes, GPFS certainly needs to behave better in this situation. We are currently working on proper support for running on newer hardware that supports Superuser Mode Access Prevention (SMAP) instructions. I believe those are new to Broadwell CPUs, but there's some confusing info out there, I'm not positive what the deal is with Haswell. For the time being, booting with the "nosmap" kernel parameter is the workaround, but you're absolutely correct, the code needs to fail more gracefully when SMAP is enabled. We'll fix that. The current FAQ structure is, without question, suboptimal. We're looking for a better format to present this information, along the lines of more modern approaches like a structured Knowledge Base. The problem is recognized, on our end, but we've been having hard time making forward progress on this. yuri From: "Sanchez, Paul" To: "gpfsug main discussion list (gpfsug-discuss at spectrumscale.org)" , Date: 06/03/2016 06:38 AM Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 on Haswell/Broadwell? Sent by: gpfsug-discuss-bounces at spectrumscale.org After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding ?nosmap? to the kernel command line in grub prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.) From https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html, section 3.2: Note: In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor Disable the Supervisor Mode Access Prevention (smap) kernel parameter Reboot the RHEL 7 node before using GPFS Some observations worth noting: 1. We?ve been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs? 2. It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without ?nosmap?. Instead, an error message describing the fix would have made my day. 3. I?m going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. Thanks, Paul Sanchez _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 8 15:50:10 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 8 Jun 2016 14:50:10 +0000 Subject: [gpfsug-discuss] GPFS Tuning Wiki Page Message-ID: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> Hi All, https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Linux%20Tuning The above Wiki page describes itself as "GPFS network tuning parameters for large clusters on Linux.? How does GPFS multi-cluster play into this equation? For example, we?ve got a ~700 node traditional HPC cluster where these parameters would obviously come into play. But we?ve also got another 9 node GPFS cluster that is mainly accessed via CNFS and CIFS from labs around campus. However, it?s filesystem is also natively GPFS mounted on the HPC cluster via GPFS multi-cluster. Do I therefore need to apply the tuning referenced in the Wiki page above to the 9 node cluster? Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Jun 8 21:31:04 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 8 Jun 2016 22:31:04 +0200 Subject: [gpfsug-discuss] GPFS Tuning Wiki Page In-Reply-To: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> References: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> Message-ID: An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Thu Jun 9 16:21:07 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Thu, 9 Jun 2016 16:21:07 +0100 Subject: [gpfsug-discuss] Dry-Run functionality Message-ID: <57598963.3040706@pixitmedia.com> Hello all, I wonder if you could have your take on your ideal dry-run functionality. Post London User Group, we had much feedback regarding the need for dry-run functionality. Therefore we have implemented dry-run functionality for our GPFS/Scale Python API across all calls. API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development Pixit Media & ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Luke.Raimbach at crick.ac.uk Thu Jun 9 16:26:49 2016 From: Luke.Raimbach at crick.ac.uk (Luke Raimbach) Date: Thu, 9 Jun 2016 15:26:49 +0000 Subject: [gpfsug-discuss] Number of Inodes in chained AFM Caches Message-ID: Hi All, We have a situation where we are chaining AFM caches together: local update on the end of a read-only cache of a non GPFS NFS share. Essentially we have an intermediate read-only cache which can pull multiple data sources in quickly over the local network at the remote site, staging the data for WAN transfer into the ultimate destination which is a local update cache. In our first set of test runs, I am seeing a large discrepancy in the number of allocated inodes in the intermediate cache and the final local-update cache. We have sized the intermediate read-only caches to contain about 10% more "Maximum number of inodes" than the source NFS share (to accommodate any growth that happens while we transfer the data). We have also size the final local-update cache to have the same maximum number of inodes. The number of allocated inodes in the intermediate read-only cache shows the number of files in the source NFS share. However in the local-update shares, I always see 1500800 allocated and am concerned that the destination is not reading all the file-system metadata from the intermediate cache. Any ideas what could be going on here? Cheers, Luke. Luke Raimbach? Senior HPC Data and Storage Systems Engineer, The Francis Crick Institute, Gibbs Building, 215 Euston Road, London NW1 2BE. E: luke.raimbach at crick.ac.uk W: www.crick.ac.uk The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE. From Robert.Oesterlin at nuance.com Sun Jun 12 01:20:23 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Sun, 12 Jun 2016 00:20:23 +0000 Subject: [gpfsug-discuss] GPFSUG Meeting at Argonne 6/10 - Presentations Message-ID: <108228BB-1A66-4CE4-86D1-0CE9B3C8196F@nuance.com> The presentations from the user group meeting on June10th at Argronne labs are now available on the spectrumscale.org site here: http://www.spectrumscale.org/presentations/ Both my myself and Kristy Kallback-Rose thank the following who helped make it happen: Argonne Labs: Bill Allcock, Gordon McPheeters IBM team: Doris Conti, Doug O'flaherty, Ulf Troppens Along with all the presenters! We'll be following up with a more detailed report in the coming days, look for that on the spectrumscale.org site too. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid SSUG-USA Co-Principal -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Mon Jun 13 17:11:07 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 12:11:07 -0400 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> Message-ID: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From cphoffma at lanl.gov Mon Jun 13 17:19:02 2016 From: cphoffma at lanl.gov (Hoffman, Christopher P) Date: Mon, 13 Jun 2016 16:19:02 +0000 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca>, <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: Hi Jaime, What in particular would you like explained more? I'd be more than happy to discuss things further. Chris ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] Sent: Monday, June 13, 2016 10:11 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS on ZFS? I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From stijn.deweirdt at ugent.be Mon Jun 13 17:25:55 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Mon, 13 Jun 2016 18:25:55 +0200 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: <90312ed5-f263-a973-c768-28f535dc1316@ugent.be> hi chris, do you have any form of HA for the zfs blockdevices/jbod (eg when a nsd reboots/breaks/...)? or do you rely on replication within GPFS? stijn On 06/13/2016 06:19 PM, Hoffman, Christopher P wrote: > Hi Jaime, > > What in particular would you like explained more? I'd be more than happy to discuss things further. > > Chris > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] > Sent: Monday, June 13, 2016 10:11 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > > I just came across this presentation on "GPFS with underlying ZFS > block devices", by Christopher Hoffman, Los Alamos National Lab, > although some of the > implementation remains obscure. > > http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf > > It would be great to have more details, in particular the possibility > of straight use of GPFS on ZFS, instead of the 'archive' use case as > described on the presentation. > > Thanks > Jaime > > > > > Quoting "Jaime Pinto" : > >> Since we can not get GNR outside ESS/GSS appliances, is anybody using >> ZFS for software raid on commodity storage? >> >> Thanks >> Jaime >> >> > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From pinto at scinet.utoronto.ca Mon Jun 13 17:26:27 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 12:26:27 -0400 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca>, <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> Hi Chris As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. How is that accomplished when you use ZFS for software raid? On page 4 I see this info, and I'm trying to interpret it: General Configuration ... * zvols * nsddevices ? echo "zdX generic? Thanks Jaime Quoting "Hoffman, Christopher P" : > Hi Jaime, > > What in particular would you like explained more? I'd be more than > happy to discuss things further. > > Chris > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] > Sent: Monday, June 13, 2016 10:11 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > > I just came across this presentation on "GPFS with underlying ZFS > block devices", by Christopher Hoffman, Los Alamos National Lab, > although some of the > implementation remains obscure. > > http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf > > It would be great to have more details, in particular the possibility > of straight use of GPFS on ZFS, instead of the 'archive' use case as > described on the presentation. > > Thanks > Jaime > > > > > Quoting "Jaime Pinto" : > >> Since we can not get GNR outside ESS/GSS appliances, is anybody using >> ZFS for software raid on commodity storage? >> >> Thanks >> Jaime >> >> > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From bsallen at alcf.anl.gov Mon Jun 13 17:33:54 2016 From: bsallen at alcf.anl.gov (Allen, Benjamin S.) Date: Mon, 13 Jun 2016 16:33:54 +0000 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> Message-ID: <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Jaime, See https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm. An example I have for add /dev/nvme* devices: * GPFS doesn't know how that /dev/nvme* are valid block devices, use a user exit script to let it know about it cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices * Edit /var/mmfs/etc/nsddevices, and add to linux section: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) do echo $dev generic done fi * Copy edited nsddevices to the rest of the nodes at the same path for host in n01 n02 n03 n04; do scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices done Ben > On Jun 13, 2016, at 11:26 AM, Jaime Pinto wrote: > > Hi Chris > > As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. > > How is that accomplished when you use ZFS for software raid? > On page 4 I see this info, and I'm trying to interpret it: > > General Configuration > ... > * zvols > * nsddevices > - echo "zdX generic" > > > Thanks > Jaime > > Quoting "Hoffman, Christopher P" : > >> Hi Jaime, >> >> What in particular would you like explained more? I'd be more than happy to discuss things further. >> >> Chris >> ________________________________________ >> From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] >> Sent: Monday, June 13, 2016 10:11 >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >> >> I just came across this presentation on "GPFS with underlying ZFS >> block devices", by Christopher Hoffman, Los Alamos National Lab, >> although some of the >> implementation remains obscure. >> >> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >> >> It would be great to have more details, in particular the possibility >> of straight use of GPFS on ZFS, instead of the 'archive' use case as >> described on the presentation. >> >> Thanks >> Jaime >> >> >> >> >> Quoting "Jaime Pinto" : >> >>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>> ZFS for software raid on commodity storage? >>> >>> Thanks >>> Jaime >>> >>> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Mon Jun 13 18:53:41 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 13 Jun 2016 13:53:41 -0400 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: How do you set the size of a ZFS file that is simulating a GPFS disk? How do "tell" GPFS about that? How efficient is this layering, compared to just giving GPFS direct access to the same kind of LUNs that ZFS is using? Hmmm... to partially answer my question, I do something similar, but strictly for testing non-performance critical GPFS functions. On any file system one can: dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a fake 3GB disk for GPFS Then use a GPFS nsd configuration record like this: %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra servers=bog-xxx Which starts out as sparse and the filesystem will dynamically "grow" as GPFS writes to it... But I have no idea how well this will work for a critical "production" system... tx, marc kaplan. From: "Allen, Benjamin S." To: gpfsug main discussion list Date: 06/13/2016 12:34 PM Subject: Re: [gpfsug-discuss] GPFS on ZFS? Sent by: gpfsug-discuss-bounces at spectrumscale.org Jaime, See https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm . An example I have for add /dev/nvme* devices: * GPFS doesn't know how that /dev/nvme* are valid block devices, use a user exit script to let it know about it cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices * Edit /var/mmfs/etc/nsddevices, and add to linux section: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) do echo $dev generic done fi * Copy edited nsddevices to the rest of the nodes at the same path for host in n01 n02 n03 n04; do scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices done Ben > On Jun 13, 2016, at 11:26 AM, Jaime Pinto wrote: > > Hi Chris > > As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. > > How is that accomplished when you use ZFS for software raid? > On page 4 I see this info, and I'm trying to interpret it: > > General Configuration > ... > * zvols > * nsddevices > - echo "zdX generic" > > > Thanks > Jaime > > Quoting "Hoffman, Christopher P" : > >> Hi Jaime, >> >> What in particular would you like explained more? I'd be more than happy to discuss things further. >> >> Chris >> ________________________________________ >> From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] >> Sent: Monday, June 13, 2016 10:11 >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >> >> I just came across this presentation on "GPFS with underlying ZFS >> block devices", by Christopher Hoffman, Los Alamos National Lab, >> although some of the >> implementation remains obscure. >> >> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >> >> It would be great to have more details, in particular the possibility >> of straight use of GPFS on ZFS, instead of the 'archive' use case as >> described on the presentation. >> >> Thanks >> Jaime >> >> >> >> >> Quoting "Jaime Pinto" : >> >>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>> ZFS for software raid on commodity storage? >>> >>> Thanks >>> Jaime >>> >>> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Mon Jun 13 19:02:51 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 14:02:51 -0400 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> As Marc, I also have questions related to performance. Assuming we let ZFS take care of the underlying software raid, what would be the difference between GPFS and Lustre for instance, for the "parallel serving" at scale part of the file system. What would keep GPFS from performing or functioning just as well? Thanks Jaime Quoting "Marc A Kaplan" : > How do you set the size of a ZFS file that is simulating a GPFS disk? How > do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct access > to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > > tx, marc kaplan. > > > > From: "Allen, Benjamin S." > To: gpfsug main discussion list > Date: 06/13/2016 12:34 PM > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Jaime, > > See > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > . An example I have for add /dev/nvme* devices: > > * GPFS doesn't know how that /dev/nvme* are valid block devices, use a > user exit script to let it know about it > > cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices > > * Edit /var/mmfs/etc/nsddevices, and add to linux section: > > if [[ $osName = Linux ]] > then > : # Add function to discover disks in the Linux environment. > for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) > do > echo $dev generic > done > fi > > * Copy edited nsddevices to the rest of the nodes at the same path > for host in n01 n02 n03 n04; do > scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices > done > > > Ben > >> On Jun 13, 2016, at 11:26 AM, Jaime Pinto > wrote: >> >> Hi Chris >> >> As I understand, GPFS likes to 'see' the block devices, even on a > hardware raid solution such as DDN's. >> >> How is that accomplished when you use ZFS for software raid? >> On page 4, I see this info, and I'm trying to interpret it: >> >> General Configuration >> ... >> * zvols >> * nsddevices >> - echo "zdX generic" >> >> >> Thanks >> Jaime >> >> Quoting "Hoffman, Christopher P" : >> >>> Hi Jaime, >>> >>> What in particular would you like explained more? I'd be more than > happy to discuss things further. >>> >>> Chris >>> ________________________________________ >>> From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] >>> Sent: Monday, June 13, 2016 10:11 >>> To: gpfsug main discussion list >>> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >>> >>> I just came across this presentation on "GPFS with underlying ZFS >>> block devices", by Christopher Hoffman, Los Alamos National Lab, >>> although some of the >>> implementation remains obscure. >>> >>> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >>> >>> It would be great to have more details, in particular the possibility >>> of straight use of GPFS on ZFS, instead of the 'archive' use case as >>> described on the presentation. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> Quoting "Jaime Pinto" : >>> >>>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>>> ZFS for software raid on commodity storage? >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> http://www.scinethpc.ca/testimonials >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.org >>> University of Toronto >>> 256 McCaul Street, Room 235 >>> Toronto, ON, M5T1W5 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of > Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of > Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From cphoffma at lanl.gov Mon Jun 13 19:15:37 2016 From: cphoffma at lanl.gov (Hoffman, Christopher P) Date: Mon, 13 Jun 2016 18:15:37 +0000 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> , <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> Message-ID: To specify the size of the disks GPFS uses one can use zvols. Then one can turn on the zfs setting sync=always to perform safe writes since I'm using SATA cards there is no BBU. In our testing, turning sync=on creates a 20%-30% decrease in overall throughput on writes. I do not have numbers of this setup vs hardware RAID6. Thanks, Chris ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] Sent: Monday, June 13, 2016 12:02 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS on ZFS! ... ? As Marc, I also have questions related to performance. Assuming we let ZFS take care of the underlying software raid, what would be the difference between GPFS and Lustre for instance, for the "parallel serving" at scale part of the file system. What would keep GPFS from performing or functioning just as well? Thanks Jaime Quoting "Marc A Kaplan" : > How do you set the size of a ZFS file that is simulating a GPFS disk? How > do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct access > to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > > tx, marc kaplan. > > > > From: "Allen, Benjamin S." > To: gpfsug main discussion list > Date: 06/13/2016 12:34 PM > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Jaime, > > See > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > . An example I have for add /dev/nvme* devices: > > * GPFS doesn't know how that /dev/nvme* are valid block devices, use a > user exit script to let it know about it > > cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices > > * Edit /var/mmfs/etc/nsddevices, and add to linux section: > > if [[ $osName = Linux ]] > then > : # Add function to discover disks in the Linux environment. > for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) > do > echo $dev generic > done > fi > > * Copy edited nsddevices to the rest of the nodes at the same path > for host in n01 n02 n03 n04; do > scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices > done > > > Ben > >> On Jun 13, 2016, at 11:26 AM, Jaime Pinto > wrote: >> >> Hi Chris >> >> As I understand, GPFS likes to 'see' the block devices, even on a > hardware raid solution such as DDN's. >> >> How is that accomplished when you use ZFS for software raid? >> On page 4, I see this info, and I'm trying to interpret it: >> >> General Configuration >> ... >> * zvols >> * nsddevices >> - echo "zdX generic" >> >> >> Thanks >> Jaime >> >> Quoting "Hoffman, Christopher P" : >> >>> Hi Jaime, >>> >>> What in particular would you like explained more? I'd be more than > happy to discuss things further. >>> >>> Chris >>> ________________________________________ >>> From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] >>> Sent: Monday, June 13, 2016 10:11 >>> To: gpfsug main discussion list >>> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >>> >>> I just came across this presentation on "GPFS with underlying ZFS >>> block devices", by Christopher Hoffman, Los Alamos National Lab, >>> although some of the >>> implementation remains obscure. >>> >>> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >>> >>> It would be great to have more details, in particular the possibility >>> of straight use of GPFS on ZFS, instead of the 'archive' use case as >>> described on the presentation. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> Quoting "Jaime Pinto" : >>> >>>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>>> ZFS for software raid on commodity storage? >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> http://www.scinethpc.ca/testimonials >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.org >>> University of Toronto >>> 256 McCaul Street, Room 235 >>> Toronto, ON, M5T1W5 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of > Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of > Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jonathan at buzzard.me.uk Mon Jun 13 20:05:20 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Mon, 13 Jun 2016 20:05:20 +0100 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: <575F03F0.2020605@buzzard.me.uk> On 13/06/16 18:53, Marc A Kaplan wrote: > How do you set the size of a ZFS file that is simulating a GPFS disk? > How do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct > access to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > For "testing" purposes I just create a logical volume and map it through to my bunch of GPFS KVM instances as a disk. Works a treat and SSD's are silly money these days so for testing performance is just fine. There was a 960GB SanDisk on offer for 160GBP last month. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From laurence at qsplace.co.uk Mon Jun 13 20:12:03 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Mon, 13 Jun 2016 22:12:03 +0300 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <575F03F0.2020605@buzzard.me.uk> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> <575F03F0.2020605@buzzard.me.uk> Message-ID: <9963179A-1D12-44BD-ABE7-5A607D0644D8@qsplace.co.uk> @JAB Same here passing the same LVM LV's through to multiple KVM instances works a treat for testing. -- Lauz On 13 June 2016 22:05:20 EEST, Jonathan Buzzard wrote: >On 13/06/16 18:53, Marc A Kaplan wrote: >> How do you set the size of a ZFS file that is simulating a GPFS disk? >> How do "tell" GPFS about that? >> >> How efficient is this layering, compared to just giving GPFS direct >> access to the same kind of LUNs that ZFS is using? >> >> Hmmm... to partially answer my question, I do something similar, but >> strictly for testing non-performance critical GPFS functions. >> On any file system one can: >> >> dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create >a >> fake 3GB disk for GPFS >> >> Then use a GPFS nsd configuration record like this: >> >> %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra >> servers=bog-xxx >> >> Which starts out as sparse and the filesystem will dynamically "grow" >as >> GPFS writes to it... >> >> But I have no idea how well this will work for a critical >"production" >> system... >> > >For "testing" purposes I just create a logical volume and map it >through >to my bunch of GPFS KVM instances as a disk. Works a treat and SSD's >are >silly money these days so for testing performance is just fine. There >was a 960GB SanDisk on offer for 160GBP last month. > >JAB. > >-- >Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk >Fife, United Kingdom. >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 14 21:50:08 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 14 Jun 2016 20:50:08 +0000 Subject: [gpfsug-discuss] QoS question Message-ID: Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jun 14 22:05:53 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 14 Jun 2016 14:05:53 -0700 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: GPFS proper (as opposed to GNR) isn't particularly picky about block devices. Any block device that GPFS can see, with help from an nsddevices user exit if necessary, is fair game, for those willing to blaze new trails. This applies to "real" devices, e.g. disk partitions or hardware RAID LUNs, and "virtual" ones, like software RAID devices. The device has to be capable to accepting IO requests of GPFS block size, but aside from that, Linux kernel does a pretty good job abstracting the realities of low-level implementation from the higher-level block device API. The basic problem with software RAID approaches is the lack of efficient HA. Since a given device is only visible to one node, if a node goes down, it takes the NSDs with it (as opposed to the more traditional twin-tailed disk model, when another NSD server can take over). So one would have to rely on GPFS data/metadata replication to get HA, and that is costly, in terms of disk utilization efficiency and data write cost. This is still an attractive model for some use cases, but it's not quite a one-to-one replacement for something like GNR for general use. yuri From: "Jaime Pinto" To: "gpfsug main discussion list" , Date: 06/13/2016 09:11 AM Subject: Re: [gpfsug-discuss] GPFS on ZFS? Sent by: gpfsug-discuss-bounces at spectrumscale.org I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Tue Jun 14 23:07:36 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 14 Jun 2016 18:07:36 -0400 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: Yes, in QOS for 4.2.0 there are some simple assumptions that may not make a lot of sense in some configurations, especially configurations with many (100s) of nodes mounting the same file system... You can try out what you suggested and in 4.2.0 I think it will pretty much work as you suggest -- essentially you are allocating 466 maintenance iops to every node, knowing that most of those nodes will not be using their allocation of IOPS. In later releases, you may find that we will address some of these kinds of quirks in QOS. QOS is a new feature for GPFS, and I don't think you'll find anything like it in any commercial file system offering. (Correct me with example(s) if I am wrong on this point.) So think of it as "release 1.0" (of QOS) and let us know how well it works for you and how it might be improved.... --marc of Spectrum(GP)FS From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2016 04:50 PM Subject: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Jun 15 06:54:54 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 15 Jun 2016 05:54:54 +0000 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: XFS on Irix had a feature similar to QoS, called GRIO (guaranteed rate I/O), where applications could reserve a given bandwidth. http://www.sgistuff.net/software/irixintro/documents/xfs-whitepaper.html Sounds somewhat similar to QoS, but focused on giving applications guaranteed bandwidth, not iops. -jf ons. 15. jun. 2016 kl. 00.08 skrev Marc A Kaplan : > Yes, in QOS for 4.2.0 there are some simple assumptions that may not make > a lot of sense in some configurations, especially configurations with many > (100s) of nodes mounting the same file system... You can try out what > you suggested and in 4.2.0 I think it will pretty much work as you suggest > -- essentially you are allocating 466 maintenance iops to every node, > knowing that most of those nodes will not be using their allocation of IOPS. > > In later releases, you may find that we will address some of these kinds > of quirks in QOS. > > QOS is a new feature for GPFS, and I don't think you'll find anything like > it in any commercial file system offering. (Correct me with example(s) if > I am wrong on this point.) > So think of it as "release 1.0" (of QOS) and let us know how well it works > for you and how it might be improved.... > > --marc of Spectrum(GP)FS > > > > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 06/14/2016 04:50 PM > Subject: [gpfsug-discuss] QoS question > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi All, > > We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to > dive into my first attempts at using the new QoS features. I want to make > sure I am understanding the documentation: > > "The IOPS values that you set in an mmchqos command apply to all I/O > operations that are issued by all the nodes that have the specified file > system mounted. You should adjust your allocations of IOPS accordingly. > > For example, if you 600 IOPS to the maintenance class, and there are six > nodes that have the file system mounted, then QoS allocates 100 IOPS to the > maintenance class of each node. If you then run maintenance commands that > affect only three of the nodes, the commands runs with an actual allocation > of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect > three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate > 1200 IOPS to the maintenanceclass. " > > We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret > the above assuming that I have determined that I want to allow 7,000 IOPs ? > please correct me if I?m wrong... > > 7,000 IOPs / 700 nodes would be 10 IOPs per node. > > But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that > are going to be doing the maintenance (a restripe, for example), so 7,000 / > 15 = 466.67. > > 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 > IOPs to the maintenance class? > > Thanks in advance? > > Kevin > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > > *Kevin.Buterbaugh at vanderbilt.edu* - > (615)875-9633 > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Wed Jun 15 12:03:10 2016 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 15 Jun 2016 13:03:10 +0200 Subject: [gpfsug-discuss] ISC U/G Agenda - Next Monday In-Reply-To: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com> References: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com> Message-ID: <201606151103.u5FAwsYp073979@mx0a-001b2d01.pphosted.com> Hello Everybody, here is a reminder for the User Group meeting at ISC in Frankfurt next Monday. Organized by IBM ... but same spirit as the meetings in London, New York and Chicago. See here for updated agenda and registration: http://www-05.ibm.com/de/events/isc16/ Looking forward to see many of you next Monday. Best regards, Ulf. -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Douglas O'flaherty" To: gpfsug-discuss at spectrumscale.org Date: 13.05.2016 17:02 Subject: [gpfsug-discuss] ISC U/G Agenda Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings: IBM is happy to announce the agenda for the joint IBM Spectrum Scale and Platform Computing User Group Agenda at ISC. We will finish on time to attend the opening reception. As with other user group meetings, the agenda includes user stories, updates on IBM Spectrum Scale & Platform LSF, and access to IBM experts and your peers. Please join us! To attend, please email Fabian.Beckmann at de.ibm.com so we can have an accurate count of attendees. Monday June 20, 2016 - 14:30-18:00 - Conference Room Konstant 14:30-14:40 [10 min] Welcome (Douglas o'Flaherty, IBM) 14:40-15:00 [20 min] Ten Reasons to Upgrade from GPFS 3.4 to Spectrum Scale 4.2 (Olaf Weiser, IBM) 15:00-15:30 [30 min] Shared Storage with in-memory latency: EMC DSSD D5 and IBM Spectrum Scale (Stefan Radtke, EMC) 15:30-16:00 [30 min] Workload scheduling and data management in a private cloud (Uwe Sommer, Airbus) 16:00-16:30 [30 min] Spectrum Scale site report (To be confirmed by customer) 16:30-17:00 [30 min] What's new in Platform LSF 10.1 & storage integration (Bill McMillan, IBM) 17:00-17:30 [30 min] What's new in Spectrum Scale 4.2.1 (Mathias Dietz, IBM) 17:30-18:00 [30 min] CORAL enhancements for Spectrum Scale (Sven Oehme, IBM) Looking forward to seeing you there! doug PS: IBMers can register their clients at this IBM link: https://w3-9033.ibm.com/events/ast/schedule/16isc.nsf _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Wed Jun 15 19:24:07 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Jun 2016 14:24:07 -0400 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: Thanks for that reference. Further research indicates that GRIO on XFS required Irix/SGI hardware features and GRIO is not in any recent implementations of XFS. Functionally GRIO, when it worked was kind of a logical complement or opposite of GPFS/QOS. GRIO was to be used to reserve bandwidth for critical applications; whereas GPFS 4.2/QOS primary purpose to restrict the IO bandwidth consumed by not-performance critical applications. Of course if you restrict set X to use no more than B, than you have effectively reserved TotalAvailable-B for the set ~X. It may interest you to know that the primary GPFS/QOS enforcement mechanisms are based on an implementation of Token Bucket ( wikipedia.org/wiki/Token_bucket , US Patent 9178827 ) --marc From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 06/15/2016 01:55 AM Subject: Re: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org XFS on Irix had a feature similar to QoS, called GRIO (guaranteed rate I/O), where applications could reserve a given bandwidth. http://www.sgistuff.net/software/irixintro/documents/xfs-whitepaper.html Sounds somewhat similar to QoS, but focused on giving applications guaranteed bandwidth, not iops. -jf ons. 15. jun. 2016 kl. 00.08 skrev Marc A Kaplan : Yes, in QOS for 4.2.0 there are some simple assumptions that may not make a lot of sense in some configurations, especially configurations with many (100s) of nodes mounting the same file system... You can try out what you suggested and in 4.2.0 I think it will pretty much work as you suggest -- essentially you are allocating 466 maintenance iops to every node, knowing that most of those nodes will not be using their allocation of IOPS. In later releases, you may find that we will address some of these kinds of quirks in QOS. QOS is a new feature for GPFS, and I don't think you'll find anything like it in any commercial file system offering. (Correct me with example(s) if I am wrong on this point.) So think of it as "release 1.0" (of QOS) and let us know how well it works for you and how it might be improved.... --marc of Spectrum(GP)FS From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2016 04:50 PM Subject: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 15:29:20 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 14:29:20 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? Message-ID: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 15:35:10 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 14:35:10 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: Message-ID: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> Hi All, And, of course, within 5 minutes of sending that e-mail I found the ?mmlsattr? command, which does what I need and which didn?t show up in my Google search results. I apologize for the noise? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamiedavis at us.ibm.com Fri Jun 17 16:04:17 2016 From: jamiedavis at us.ibm.com (James Davis) Date: Fri, 17 Jun 2016 15:04:17 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> References: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu>, Message-ID: An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 16:18:48 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 15:18:48 +0000 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: References: Message-ID: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Hi yet again all, Well, this has turned out to be an enlightening and surprising morning in GPFS land? What prompted my question below is this ? I am looking to use the new QoS features in GPFS 4.2. I have QoS enabled and am trying to get a baseline of IOPs so that I can determine how much I want to assign to the maintenance class (currently both maintenance and other are set to unlimited). To do this, I fired off a bonnie++ test from each of my NSD servers. The filesystem in question has two storage pools, the system pool and the capacity pool. The system pool is comprised of a couple of metadata only disks (SSD-based RAID 1 mirrors) and several data only disks (spinning HD-based RAID 6), while the capacity pool is comprised exclusively of data only disks (RAID 6). When the bonnie++?s were creating, reading, and rewriting the big file they create I was quite surprised to see mmlsqos show higher IOP?s on the capacity pool than the system pool by a factor of 10! As I was expecting those files to be being written to the system pool, this was quite surprising to me. Once I found the mmlsattr command, I ran it on one of the files being created and saw that it was indeed assigned to the capacity pool. The bonnie++?s finished before I could check the other files. I don?t have any file placement policies in effect for this filesystem, only file migration policies (each weekend any files in the system pool with an atime > 60 days get moved to the capacity pool and any files in the capacity pool with an atime < 60 days get moved to the system pool). In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.? This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don?t remember for sure) later. I was under the obviously mistaken impression that the ?first? data storage pool was the system pool, but that is clearly not correct. So my first question is, what is the definition of ?the first storage pool?? and my second question is, can the documentation be updated with the answer to my first question since it?s clearly ambiguous as written now? Thanks? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 16:21:48 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 15:21:48 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> Message-ID: <9C2AC549-B85E-40EE-92CA-98983A4B3D85@vanderbilt.edu> Hi Jamie, The much longer e-mail I just sent to the list will explain the background behind all this, but to answer your question I actually only needed to know which storage pool the file resided in. Obviously, if I know which NSDs it resides on that would tell me the storage pool. Thanks? Kevin On Jun 17, 2016, at 10:04 AM, James Davis > wrote: Hi Kevin, It looks like you can use mmlsattr to query the storage pool the file resides on. Is that all you needed to know, or did you need the specific disks on which it resides? If you're looking for the specific disks, how are you using the mmlsattr command to accomplish that? Jamie Jamie Davis GPFS Functional Verification Test (FVT) jamiedavis at us.ibm.com ----- Original message ----- From: "Buterbaugh, Kevin L" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: Re: [gpfsug-discuss] Can you query which NSDs a given file resides on? Date: Fri, Jun 17, 2016 10:36 AM Hi All, And, of course, within 5 minutes of sending that e-mail I found the ?mmlsattr? command, which does what I need and which didn?t show up in my Google search results. I apologize for the noise? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 17 16:22:44 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 11:22:44 -0400 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: Message-ID: Hint: tsdbfs More hints: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014215697 If you mess up, you didn't hear this from me ;-) From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/17/2016 10:30 AM Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamiedavis at us.ibm.com Fri Jun 17 16:50:26 2016 From: jamiedavis at us.ibm.com (James Davis) Date: Fri, 17 Jun 2016 15:50:26 +0000 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu>, Message-ID: An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Fri Jun 17 16:55:21 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 17 Jun 2016 16:55:21 +0100 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: <1466178921.3773.107.camel@buzzard.phy.strath.ac.uk> On Fri, 2016-06-17 at 15:18 +0000, Buterbaugh, Kevin L wrote: > Hi yet again all, > > > Well, this has turned out to be an enlightening and surprising morning > in GPFS land? > > > What prompted my question below is this ? I am looking to use the new > QoS features in GPFS 4.2. I have QoS enabled and am trying to get a > baseline of IOPs so that I can determine how much I want to assign to > the maintenance class (currently both maintenance and other are set to > unlimited). To do this, I fired off a bonnie++ test from each of my > NSD servers. > > > The filesystem in question has two storage pools, the system pool and > the capacity pool. The system pool is comprised of a couple of > metadata only disks (SSD-based RAID 1 mirrors) and several data only > disks (spinning HD-based RAID 6), while the capacity pool is comprised > exclusively of data only disks (RAID 6). > My preference in these scenarios is to have the system pool for metadata only. Then have a "fast" pool and a "slow" pool. Pool names more than 7 characters is a recipe for backup headaches with TSM at least, so "fast" and "slow" are deliberately chosen as names :) That way you can tweak block sizes more optimally. > > When the bonnie++?s were creating, reading, and rewriting the big file > they create I was quite surprised to see mmlsqos show higher IOP?s on > the capacity pool than the system pool by a factor of 10! As I was > expecting those files to be being written to the system pool, this was > quite surprising to me. Once I found the mmlsattr command, I ran it > on one of the files being created and saw that it was indeed assigned > to the capacity pool. The bonnie++?s finished before I could check > the other files. > > > I don?t have any file placement policies in effect for this > filesystem, only file migration policies (each weekend any files in > the system pool with an atime > 60 days get moved to the capacity pool > and any files in the capacity pool with an atime < 60 days get moved > to the system pool). Note due to the way applications handle saving files by creating a new file, moving files from slow to fast disk is mostly a waste of effort in my experience. By the time you do that, 99% of the time the user has finished with the file anyway. When they haven't generally the slow disk has gazillions of free IOPs anyway. > > In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS > file system does not have a placement policy installed, all the data > is stored in the first data storage pool.? > My understanding is that no placement policy and it can go anywhere, where anywhere is the next available NSD with free space. I found that you needed to do something like this at the end of your policy file for the default placement. /* move files over 1GB onto nearline storage and keep them there */ RULE 'big' MIGRATE FROM POOL 'system' TO POOL 'nearline' WHERE FILE_SIZE > 1073741824 /* migrate old files to the RAID6 disks to keep the RAID1 disks free */ RULE 'ilm' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(weighting) TO POOL 'slow' /* by default new files to the fast RAID1 disk unless full, then to RAID6 */ RULE 'new' SET POOL 'fast' LIMIT(95) RULE 'spillover' SET POOL 'slow' You have to stop allocation to the fast disk from ever approaching 100%. Otherwise a you get disk full errors on new file writes. Imagine you create a new file when there is 1KB of space free in the fast pool, then try and write 10KB of data; it will fail with an error. So first rule forces large files to the slow pool anyway (you can tweak this size to suite your environment) but it must be significantly less than the the space free when you hit 95% full on the fast disk. You then need to have a rule to kick the tiering migration in lower than when you start allocating to the slow pool. Finally you allocation new files to the fast disk till it gets quite full, before spilling over to the slow pool (which one presumes has loads of free capacity) if the fast pool fills up. Not shown you ideally need to kick in tiering migration every night so that it does not run during the day when users might notice the performance degradation. Ideally you want to have enough free space come the morning in your fast disk pool that all new files can be created there during the day. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From makaplan at us.ibm.com Fri Jun 17 16:56:59 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 11:56:59 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (doh!) In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: Thanks for asking! Prior to Release 4.2 system pool was the default pool for storing file data and you had to write at least one policy SET POOL rule to make use of any NSDs that you assigned to some other pool. Now if a file system is formatted or upgraded to level 4.2 or higher, and there is a data pool defined, and there are no SET POOL policy rules, then the "first" such data pool becomes the default storage pool for file data. "metadata" is always stored in system pool. So how is the "first" data pool determined? It's usually the first data pool added to the file system. For many customers and installations there is only one such pool, so no problem - and this is exactly what they wanted all along (since we introduced "pools" in release 3.1) We received complaints over the years along the lines: "Heh! Why the heck do you think I made a data pool? I don't want to know about silly SET POOL rules and yet another mm command (mmchpolicy)! Just do it!" Well in 4.2 we did. If you look into the /var/adm/ras/mmfs.log.* file(s) you will see that during mmmount 4.2 will tell you.... Fri Jun 17 09:31:26.637 2016: [I] Command: mount yy Fri Jun 17 09:31:27.625 2016: [I] Loaded policy 'for file system yy': Parsed 4 policy rules. Fri Jun 17 09:31:27.626 2016: Policy has no storage pool placement rules. Fri Jun 17 09:31:27.627 2016: [I] Data will be stored in pool 'xtra'. Notice we DID NOT change the behavior for file systems at level 4.1 or prior, even when you upgrade the software. But when you upgrade the file system to 4.2 (for example to use QOS) ... From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/17/2016 11:19 AM Subject: [gpfsug-discuss] Initial file placement Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi yet again all, Well, this has turned out to be an enlightening and surprising morning in GPFS land? What prompted my question below is this ? I am looking to use the new QoS features in GPFS 4.2. I have QoS enabled and am trying to get a baseline of IOPs so that I can determine how much I want to assign to the maintenance class (currently both maintenance and other are set to unlimited). To do this, I fired off a bonnie++ test from each of my NSD servers. The filesystem in question has two storage pools, the system pool and the capacity pool. The system pool is comprised of a couple of metadata only disks (SSD-based RAID 1 mirrors) and several data only disks (spinning HD-based RAID 6), while the capacity pool is comprised exclusively of data only disks (RAID 6). When the bonnie++?s were creating, reading, and rewriting the big file they create I was quite surprised to see mmlsqos show higher IOP?s on the capacity pool than the system pool by a factor of 10! As I was expecting those files to be being written to the system pool, this was quite surprising to me. Once I found the mmlsattr command, I ran it on one of the files being created and saw that it was indeed assigned to the capacity pool. The bonnie++?s finished before I could check the other files. I don?t have any file placement policies in effect for this filesystem, only file migration policies (each weekend any files in the system pool with an atime > 60 days get moved to the capacity pool and any files in the capacity pool with an atime < 60 days get moved to the system pool). In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.? This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don?t remember for sure) later. I was under the obviously mistaken impression that the ?first? data storage pool was the system pool, but that is clearly not correct. So my first question is, what is the definition of ?the first storage pool?? and my second question is, can the documentation be updated with the answer to my first question since it?s clearly ambiguous as written now? Thanks? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L < Kevin.Buterbaugh at Vanderbilt.Edu> wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 17 17:10:55 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 12:10:55 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (doh!) (DOH!) 4.1.1 In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: OOPS... The story I gave was almost correct. The cut over was at 4.1.1 (after 4.1 before 4.2) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Jun 17 17:20:22 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 12:20:22 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (DOC!) In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: (Doc?... I ain't about to read a manual! But just to cover our bases we did stick it into the 4.1.1 pubs) Policy improvements: This release includes the following policy improvements: ... (summary of changes, page xv) ... Implicit SET POOL 'first-data-pool' rule For file systems that are at or have been upgraded to 4.1.1, the system recognizes that, even if no policy rules have been installed to a file system by mmchpolicy , data files should be stored in a non-system pool if available (rather than in the system pool, which is the default for earlier releases). For more information, see the following: Information Lifecycle Management chapter in the IBM Spectrum Scale: Advanced Administration Guide ... ( mmchpolicy man page 298) ... For file systems that are at or have been upgraded to 4.1.1 or later: If there are no SET POOL policy rules installed to a file system by mmchpolicy, the system acts as if the single rule SET POOL 'first-data-pool' is in effect, where first-data-pool is the firstmost non-system pool that is available for file data storage, if such a non-system pool is available. (?Firstmost? is the first according to an internal index of all pools.) However, if there are no policy rules installed and there is no non-system pool, the system acts as if SET POOL 'system' is in effect. This change applies only to file systems that were created at or upgraded to 4.1.1. Until a file system is upgraded, if no SET POOL rules are present (set by mmchpolicy) for the file system, all data will be stored in the 'system' pool. For information on GPFS policies, see the IBM Spectrum Scale: Advanced Administration Guide. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 17:33:39 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 16:33:39 +0000 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (DOC!) In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: <78BBA4B7-505F-4D21-BA14-BC430D93F993@vanderbilt.edu> Hi Marc, Read every last one of the thousands of pages of documentation that comprises the GPFS documentation set before doing any GPFS upgrades ? yes, I apparently *do* need to do that! ;-) The paragraph that I find ambiguous can be found on page 20 of the GPFS 4.2 Advanced Administration Guide (3rd paragraph from bottom): The placement policy defining the initial placement of newly created files and the rules for placement of * | restored data must be installed into GPFS with the mmchpolicy command. If a GPFS file system does not * | have a placement policy installed, all the data is stored in the first data storage pool. Only one placement policy can be installed at a time. If you switch from one placement policy to another, or make changes to a placement policy, that action has no effect on existing files. However, newly created files are always placed according to the currently installed placement policy. In my opinion, it should at least contain a pointer the pages you reference below. Preferentially, it would contain the definition of ?first data storage pool?. Thanks? Kevin On Jun 17, 2016, at 11:20 AM, Marc A Kaplan > wrote: (Doc?... I ain't about to read a manual! But just to cover our bases we did stick it into the 4.1.1 pubs) Policy improvements: This release includes the following policy improvements: ... (summary of changes, page xv) ... Implicit SET POOL 'first-data-pool' rule For file systems that are at or have been upgraded to 4.1.1, the system recognizes that, even if no policy rules have been installed to a file system by mmchpolicy, data files should be stored in a non-system pool if available (rather than in the system pool, which is the default for earlier releases). For more information, see the following: Information Lifecycle Management chapter in the IBM Spectrum Scale: Advanced Administration Guide ... ( mmchpolicy man page 298) ... For file systems that are at or have been upgraded to 4.1.1 or later: If there are no SET POOL policy rules installed to a file system by mmchpolicy, the system acts as if the single rule SET POOL 'first-data-pool' is in effect, where first-data-pool is the firstmost non-system pool that is available for file data storage, if such a non-system pool is available. (?Firstmost? is the first according to an internal index of all pools.) However, if there are no policy rules installed and there is no non-system pool, the system acts as if SET POOL 'system' is in effect. This change applies only to file systems that were created at or upgraded to 4.1.1. Until a file system is upgraded, if no SET POOL rules are present (set by mmchpolicy) for the file system, all data will be stored in the 'system' pool. For information on GPFS policies, see the IBM Spectrum Scale: Advanced Administration Guide. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 14:45:34 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 13:45:34 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions Message-ID: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don't know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set "force user = root" all the snapshots are available. I think the answer is SAMBA, right? We're running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 14:49:36 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 13:49:36 +0000 Subject: [gpfsug-discuss] Dry-Run functionality In-Reply-To: <57598963.3040706@pixitmedia.com> References: <57598963.3040706@pixitmedia.com> Message-ID: IMO I?d want to see an explanation of what will happen in plain English. ?This will create a policy that moves any file bigger than 10MB and older than 30 days to storage pool XYZ?. You might be able to tell I?ve never used GPFS policies except for initial placement so the above example might not be possible anyway ? I?m very used to the Powershell switch ?-whatif? which basically said ?I am going to do the following:? Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jez Tucker Sent: 09 June 2016 16:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] Dry-Run functionality Hello all, I wonder if you could have your take on your ideal dry-run functionality. Post London User Group, we had much feedback regarding the need for dry-run functionality. Therefore we have implemented dry-run functionality for our GPFS/Scale Python API across all calls. API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development Pixit Media & ArcaStream www.pixitmedia.com www.arcastream.com [Image removed by sender.] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD000.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD000.jpg URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Jun 20 15:56:27 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 20 Jun 2016 14:56:27 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions In-Reply-To: References: Message-ID: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> Hi Richard, I can?t answer your question but I can tell you that we have experienced either the exact same thing you are or something very similar. It occurred for us after upgrading from GPFS 3.5 to 4.1.0.8 and it persists even after upgraded to GPFS 4.2.0.3 and the very latest sernet-samba. And to be clear, when we upgraded from GPFS 3.5 to 4.1 we did *not* upgrade SAMBA versions at that time. Therefore, I believe that something changed in GPFS. That doesn?t mean it?s GPFS? fault, of course. SAMBA may have been relying on a bugundocumented feature in GPFS that IBM fixed for all I know, and I?m obviously speculating here. The problem we see is that the .snapshots directory in each folder can be cd?d to but is empty. The snapshots are all there, however, if you: cd //.snapshots//rest/of/path/to/folder/in/question This obviously prevents users from being able to do their own recovery of files unless you do something like what you describe, which we are unwilling to do for security reasons. We have a ticket open with DDN? Kevin On Jun 20, 2016, at 8:45 AM, Sobey, Richard A > wrote: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don?t know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set ?force user = root? all the snapshots are available. I think the answer is SAMBA, right? We?re running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 16:03:28 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 15:03:28 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions In-Reply-To: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> References: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> Message-ID: Thanks Kevin. We are upgrading to GPFS 4.2 and CES in a few weeks but our customers have come to like previous versions and indeed it is sort of a selling point for us. Samba is the only thing we?ve changed recently after the badlock debacle so I?m tempted to blame that, but who knows. If (when) I find out I?ll let everyone know. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: 20 June 2016 15:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Snapshots / Windows previous versions Hi Richard, I can?t answer your question but I can tell you that we have experienced either the exact same thing you are or something very similar. It occurred for us after upgrading from GPFS 3.5 to 4.1.0.8 and it persists even after upgraded to GPFS 4.2.0.3 and the very latest sernet-samba. And to be clear, when we upgraded from GPFS 3.5 to 4.1 we did *not* upgrade SAMBA versions at that time. Therefore, I believe that something changed in GPFS. That doesn?t mean it?s GPFS? fault, of course. SAMBA may have been relying on a bugundocumented feature in GPFS that IBM fixed for all I know, and I?m obviously speculating here. The problem we see is that the .snapshots directory in each folder can be cd?d to but is empty. The snapshots are all there, however, if you: cd //.snapshots//rest/of/path/to/folder/in/question This obviously prevents users from being able to do their own recovery of files unless you do something like what you describe, which we are unwilling to do for security reasons. We have a ticket open with DDN? Kevin On Jun 20, 2016, at 8:45 AM, Sobey, Richard A > wrote: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don?t know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set ?force user = root? all the snapshots are available. I think the answer is SAMBA, right? We?re running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 20 16:03:50 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Jun 2016 11:03:50 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 20 16:03:50 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Jun 2016 11:03:50 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Mon Jun 20 16:10:13 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Mon, 20 Jun 2016 15:10:13 +0000 Subject: [gpfsug-discuss] ESS GL6 Message-ID: Couple of questions regarding Spectrum Scale 4.2 and ESS. We recently got our ESS delivered and are putting it in production this week. Previous to ESS we ran GPFS 3.5 and IBM DCS3700 storage arrays. My question about ESS and Spectrum Scale has to do with querying available free space and adding capacity to existing file system. In the old days of GPFS 3.5 I would create LUNs on 3700, zone them to appropriate hosts, and then see them as multipath devices on NSD servers. After that, I would create NSD disks and add them to the filesystem. With the ESS, however, I don't think the process is quite the same. IBM tech that was here installing the system has created all the "LUNs" or the equivalent in the ESS system. How do you I query what space is available to add to the existing filesystems, and then how do you actually add space? I am reading ESS RedBook but the answers are not obvious. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Jun 20 16:25:31 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 20 Jun 2016 17:25:31 +0200 Subject: [gpfsug-discuss] ESS GL6 In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Jun 20 17:29:15 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 20 Jun 2016 16:29:15 +0000 Subject: [gpfsug-discuss] ESS GL6 In-Reply-To: References: Message-ID: "mmlsrecoverygroup $name -L" will tell you how much raw capacity is left in a recoverygroup. You will then need to create a vdisk stanza file where you specify the capacity, blocksize, raidcode, etc. for the vdisk you want. (check "man mmcrvdisk") Then "mmcrvdisk stanzafile" to create the vdisk, and "mmcrnsd stanzafile" to create the NSDs. From then on it's standard gpfs. -jf man. 20. jun. 2016 kl. 17.25 skrev Olaf Weiser : > Hi Damir, > mmlsrecovergroup --> will show your RG > > mmlsrecoverygroup RG -L .. will provide capacity information > > or .. you can use the GUI > > with ESS / GNR , there's no need any more to create more than one > vdisk(=nsd) per RG for a pool > > a practical approach/example for you > so a file system consists of > 1 vdisk(NSD) for MetaData , RAID: 4WR , BS 1M in RG"left" > 1 vdisk(NSD) for MetaData , Raid : 4WR, BS 1M in RG "right" > 1 vdisk (NSD) for data , 8+3p , BS 1...16M .. depends on your > data/workload in RG "left" > 1 vdisk (NSD) for data , 8+3p , BS 1...16M .. depends on your > data/workload in RG "right" > > so 4 NSDs to provide everything you need to serve a file system .. > > > the size of the vdisks can be up to half of the capacity of your RG > > please note: if you come from an existing environment , and the file > system should be migrated to ESS (online) , you might hit some limitations > like > - blocksize (can not be changed) > - disk size.. depending on the existing storage pools/disk sizes.... > > > have fun > cheers > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert > Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, > HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Damir Krstic > To: gpfsug main discussion list > Date: 06/20/2016 05:10 PM > Subject: [gpfsug-discuss] ESS GL6 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Couple of questions regarding Spectrum Scale 4.2 and ESS. We recently got > our ESS delivered and are putting it in production this week. Previous to > ESS we ran GPFS 3.5 and IBM DCS3700 storage arrays. > > My question about ESS and Spectrum Scale has to do with querying available > free space and adding capacity to existing file system. > > In the old days of GPFS 3.5 I would create LUNs on 3700, zone them to > appropriate hosts, and then see them as multipath devices on NSD servers. > After that, I would create NSD disks and add them to the filesystem. > > With the ESS, however, I don't think the process is quite the same. IBM > tech that was here installing the system has created all the "LUNs" or the > equivalent in the ESS system. How do you I query what space is available to > add to the existing filesystems, and then how do you actually add space? > > I am reading ESS RedBook but the answers are not obvious. > > Thanks, > Damir _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Mon Jun 20 20:35:54 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Mon, 20 Jun 2016 19:35:54 +0000 Subject: [gpfsug-discuss] easiest way of restarting Spectrum Scale 4.2 web interface Message-ID: My ESS/Spectrum Scale web interface is spinning with "loading" screen without ever prompting me for username and password. I was able to log in early last week but something has happened since last week to make it stop working. Is there a quick way of reloading web interface to see if that fixes it before I submit help ticket with IBM? Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Jun 20 20:38:03 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 20 Jun 2016 19:38:03 +0000 Subject: [gpfsug-discuss] easiest way of restarting Spectrum Scale 4.2 web interface In-Reply-To: References: Message-ID: "systemctl restart gpfsgui" -jf man. 20. jun. 2016 kl. 21.36 skrev Damir Krstic : > My ESS/Spectrum Scale web interface is spinning with "loading" screen > without ever prompting me for username and password. I was able to log in > early last week but something has happened since last week to make it stop > working. > > Is there a quick way of reloading web interface to see if that fixes it > before I submit help ticket with IBM? > > Thanks, > Damir > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaurang.tapase at in.ibm.com Wed Jun 22 06:25:47 2016 From: gaurang.tapase at in.ibm.com (Gaurang Tapase) Date: Wed, 22 Jun 2016 10:55:47 +0530 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Message-ID: Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 22 11:33:40 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 22 Jun 2016 10:33:40 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: Message-ID: Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) -------------------------------------------------------------------------Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 22 11:39:06 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 22 Jun 2016 10:39:06 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: Message-ID: Does anyone in the field have much experience with using file heat for migration, whether for object or more generically? In particular using policies to move files both ways dependant on recent usage patterns. And also if you ever move files to colder storage without necessarily waiting until the files are say over 7 days old, since you know they are not going to be used for a while. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 12:34:11, daniel.kidger at uk.ibm.com wrote: From: daniel.kidger at uk.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: Date: 22 Jun 2016 12:34:11 Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AUUnless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Wed Jun 22 12:30:10 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 22 Jun 2016 11:30:10 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Message-ID: <1F32A0BF-7B3B-4407-B0D0-1F05A4B144ED@nuance.com> Hi Daniel I've done a fair amount of work with GPFS "File Heat" and policy migration here at Nuance. Let me know if you want to discuss it a bit more. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid robert.oesterlin @ nuance.com From: on behalf of Daniel Kidger Reply-To: gpfsug main discussion list Date: Wednesday, June 22, 2016 at 5:39 AM To: gpfsug main discussion list Cc: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Does anyone in the field have much experience with using file heat for migration, whether for object or more generically? In particular using policies to move files both ways dependant on recent usage patterns. And also if you ever move files to colder storage without necessarily waiting until the files are say over 7 days old, since you know they are not going to be used for a while. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse ________________________________ On 22 Jun 2016, 12:34:11, daniel.kidger at uk.ibm.com wrote: From: daniel.kidger at uk.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: Date: 22 Jun 2016 12:34:11 Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse ________________________________ On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Wed Jun 22 13:21:54 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Wed, 22 Jun 2016 13:21:54 +0100 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: References: Message-ID: <1466598114.3773.201.camel@buzzard.phy.strath.ac.uk> On Wed, 2016-06-22 at 10:39 +0000, Daniel Kidger wrote: > Does anyone in the field have much experience with using file heat for > migration, whether for object or more generically? In particular using > policies to move files both ways dependant on recent usage patterns. > In my experience moving files from slow to fast storage (I am ignoring a tape based or other really slow layer here) was generally a waste of time for three reasons. Firstly by the time you notice that the file is in use again 99% of the time user has stopped using it anyway so there is no advantage to moving it back to faster storage. You would need to scan the file system more than once a day to capture more reused files in my experience. Secondly if the data is modified and saved due to the way most software handles this you end up with a "new" file which will land in the fast storage anyway. That is most software on a save creates a new temporary file first with the new version of the file, renames the old file to something temporary, renames the new file to the right file name and then finally deletes the old version of file. That way if the program crashes somewhere in the save there will always be a good version of the file to go back to. Thirdly the slower tier (I am thinking along the lines of large RAID6 or equivalent devices) generally has lots of spare IOPS capacity anyway. Consequently the small amount or "revisited" data that does remain in use beyond a day is not going to have a significant impact. I measured around 10 IOPS on average on the slow disk in a general purpose file system. The only time it broke this was when a flush of data form the faster tier arrived at which point they would peg out at around 120 IOPS which is what you would expect for a 7200 RPM disk. > > And also if you ever move files to colder storage without necessarily > waiting until the files are say over 7 days old, since you know they > are not going to be used for a while. > I would do things like punt .iso images directly to slower storage. Those sorts of files generally don't benefit from having low seek times which is what your fast disk pools give you. I would suggest that video files like MPEG and MP4 could be potentially also be treated similarly. So a rule like the following /* force ISO images onto nearline storage */ RULE 'iso' SET POOL 'slow' WHERE LOWER(NAME) LIKE '%.iso' What you might punt to slower storage will likely depend heavily on what you file system is being used for. You can of course also use this to "discourage" your storage from being used for "personal" use by giving certain file types lower performance. So a rule like the following puts all the music files on slower storage. /* force MP3's and the like onto nearline storage forever */ RULE 'mp3' SET POOL 'slow' WHERE LOWER(NAME) LIKE '%.mp3' OR LOWER(NAME) LIKE '%.m4a' OR LOWER(NAME) LIKE '%.wma' JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From chekh at stanford.edu Fri Jun 24 19:09:59 2016 From: chekh at stanford.edu (Alex Chekholko) Date: Fri, 24 Jun 2016 11:09:59 -0700 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: References: Message-ID: <576D7777.50609@stanford.edu> On 06/21/2016 10:25 PM, Gaurang Tapase wrote: > Hello, > > Based on inputs from some of the Spectrum Scale users, sharing a blog > link for "Automatic tiering of object data based on object heat" by Smita. > https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ > Thanks for the blog post. I wanted to post my experiences: Note "sata0" is really NL-SAS. [root at scg-gs0 ~]# df -h|grep gsfs /dev/gsfs0 4.4P 3.1P 1.4P 70% /srv/gsfs0 [root at scg-gs0 ~]# df -i|grep gsfs /dev/gsfs0 314572800 234237984 80334816 75% /srv/gsfs0 [root at scg-gs0 ~]# mmlspool gsfs0 Storage pools in file system at '/srv/gsfs0': Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB) system 0 256 KB yes yes 1887436800 118287616 ( 6%) 1887436800 328248832 ( 17%) sas0 65537 256 KB yes no 259795189760 59495012864 ( 23%) 0 0 ( 0%) sata0 65538 256 KB yes no 4523875631104 1405369561600 ( 31%) 0 0 ( 0%) [root at scg-gs0 ~]# mmlspolicy gsfs0 -L RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' [root at scg-gs0 ~]# cat policy_gpool.txt rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' LIMIT(80) THEN 'sata0' rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT) [root at scg-gs0 ~]# mmlsconfig|grep -i heat fileHeatPeriodMinutes 1440 fileHeatLossPercent 1 The first thing to know is that the MIGRATE rules only run when you run 'mmapplypolicy', even is you have the migrate policy loaded. So you have to decide when/how to do that. The filesystem scan takes a while; here are the parameters that let a full scan of our filesystem complete in under 10mins (8 I/O servers, ~300M inodes): mmapplypolicy gsfs0 -P policy.txt -N scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g /srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4 The issues we have are: 1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD, 259TB 10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% ?fast?, 94.57% ?slow?) 2) the client I/O can max out the IOPS in the "slow" pool and in the "fast" pool The original idea was to have all client I/O go to the 10K pool, then migrate least-used data to slow pool. Unfortunately our fast pool fills up too quickly and then GPFS can't empty it quickly enough into the slow pool. So I end up switching the placement policy to put files into the slow pool. And then emptying the fast pool into the slow pool doesn't work too well because the slow pool is already maxed out on IOPS just handling the workload. So moving the data from fast to slow takes weeks and impacts performance. The original intention was: [root at scg-gs0 ~]# cat policy_placement.txt RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%' RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' I would recommend that your total workload fits in both capacity and performance into your "primary" aka "fast" pool. I also just learned recently of the LIMIT parameter for placement, e.g. RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' LIMIT (95) RULE 'spillover' SET POOL 'sata0' So that would help with the pool not filling, but won't help with not having enough I/O for migration+clients. Regards, -- Alex Chekholko chekh at stanford.edu From kraemerf at de.ibm.com Sun Jun 26 12:04:42 2016 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Sun, 26 Jun 2016 13:04:42 +0200 Subject: [gpfsug-discuss] FIY - Efficient data tiering with the Swift High Latency Middleware (SwiftHLM) Message-ID: https://developer.ibm.com/open/2016/06/21/efficient-data-tiering-swift-high-latency-middleware-swifthlm/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Sun Jun 26 16:22:46 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Sun, 26 Jun 2016 15:22:46 +0000 Subject: [gpfsug-discuss] verbs rdma errors in logs Message-ID: We recently enabled verbs/rdma on our IB network (previously we used IPoIB exclusively) and now are getting all sorts of errors/warning in logs: Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA rdma read error IBV_WC_RETRY_EXC_ERR to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 vendor_err 129 Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 1589 Jun 25 20:40:05 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.124.12 (qnode4054-ib0.quest) on mlx5_0 port 1 fabnum 0 index 1417 Jun 25 18:30:01 ems1 root: mmfs: [N] VERBS RDMA closed connection to 172.41.130.131 (qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 195 Jun 25 18:28:23 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.130.131 (qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 1044 Something to note (not sure if this is important or not) is that our ESS storage cluster and our login nodes are in connected mode with 64K MTU and all compute nodes are in datagram mode with 2.4K MTU. Are these messages something to be concerned about? Cluster seems to be performing well and although there are some node ejections, they do not seem higher than before we turned on verbs/rdma. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sun Jun 26 17:14:10 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Sun, 26 Jun 2016 16:14:10 +0000 Subject: [gpfsug-discuss] verbs rdma errors in logs References: [gpfsug-discuss] verbs rdma errors in logs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101C368BF@NDMSMBX404.ndc.nasa.gov> Damir, We see similar things in our environment (3.5k nodes) that seem to correlate with GPFS recovery events. I did some digging at it seemed to me that these errors more or less mean the other side of the VERBS connection hung up on the other. The message format seems a little alarming but I think it's innocuous. I'm curious to hear what others have to say. -Aaron From: Damir Krstic Sent: 6/26/16, 11:23 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] verbs rdma errors in logs We recently enabled verbs/rdma on our IB network (previously we used IPoIB exclusively) and now are getting all sorts of errors/warning in logs: Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA rdma read error IBV_WC_RETRY_EXC_ERR to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 vendor_err 129 Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 1589 Jun 25 20:40:05 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.124.12 (qnode4054-ib0.quest) on mlx5_0 port 1 fabnum 0 index 1417 Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 195 Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 1044 Something to note (not sure if this is important or not) is that our ESS storage cluster and our login nodes are in connected mode with 64K MTU and all compute nodes are in datagram mode with 2.4K MTU. Are these messages something to be concerned about? Cluster seems to be performing well and although there are some node ejections, they do not seem higher than before we turned on verbs/rdma. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 27 15:49:27 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 27 Jun 2016 10:49:27 -0400 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: <576D7777.50609@stanford.edu> References: <576D7777.50609@stanford.edu> Message-ID: I'd like to see comments and/or questions and/or suggestions from any others who have implemented, deployed or seriously planned using FILE_HEAT and/or GROUP POOL. We implemented this a few years ago. Since we haven't had much feedback, it's either perfect or ... ? ;-) Alex, my reading of your story is that your hardware does not match your requirements, capacity, speeds, feeds. One new possibly helpful improvement is that SS release 4.2 includes QOSio which allows one to control the IOPs consumed by migration. So you can control the performance "hit" of running mmapplypolicy. You can even control it on-the-fly without killing a long running job -- for example you can throttle it back to 100iops during prime-time day shift operations -- marc of GPFS. From: Alex Chekholko To: gpfsug-discuss at spectrumscale.org Date: 06/24/2016 02:10 PM Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Sent by: gpfsug-discuss-bounces at spectrumscale.org On 06/21/2016 10:25 PM, Gaurang Tapase wrote: > Hello, > > Based on inputs from some of the Spectrum Scale users, sharing a blog > link for "Automatic tiering of object data based on object heat" by Smita. > https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ > Thanks for the blog post. I wanted to post my experiences: Note "sata0" is really NL-SAS. [root at scg-gs0 ~]# df -h|grep gsfs /dev/gsfs0 4.4P 3.1P 1.4P 70% /srv/gsfs0 [root at scg-gs0 ~]# df -i|grep gsfs /dev/gsfs0 314572800 234237984 80334816 75% /srv/gsfs0 [root at scg-gs0 ~]# mmlspool gsfs0 Storage pools in file system at '/srv/gsfs0': Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB) system 0 256 KB yes yes 1887436800 118287616 ( 6%) 1887436800 328248832 ( 17%) sas0 65537 256 KB yes no 259795189760 59495012864 ( 23%) 0 0 ( 0%) sata0 65538 256 KB yes no 4523875631104 1405369561600 ( 31%) 0 0 ( 0%) [root at scg-gs0 ~]# mmlspolicy gsfs0 -L RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' [root at scg-gs0 ~]# cat policy_gpool.txt rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' LIMIT(80) THEN 'sata0' rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT) [root at scg-gs0 ~]# mmlsconfig|grep -i heat fileHeatPeriodMinutes 1440 fileHeatLossPercent 1 The first thing to know is that the MIGRATE rules only run when you run 'mmapplypolicy', even is you have the migrate policy loaded. So you have to decide when/how to do that. The filesystem scan takes a while; here are the parameters that let a full scan of our filesystem complete in under 10mins (8 I/O servers, ~300M inodes): mmapplypolicy gsfs0 -P policy.txt -N scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g /srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4 The issues we have are: 1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD, 259TB 10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% ?fast?, 94.57% ?slow?) 2) the client I/O can max out the IOPS in the "slow" pool and in the "fast" pool The original idea was to have all client I/O go to the 10K pool, then migrate least-used data to slow pool. Unfortunately our fast pool fills up too quickly and then GPFS can't empty it quickly enough into the slow pool. So I end up switching the placement policy to put files into the slow pool. And then emptying the fast pool into the slow pool doesn't work too well because the slow pool is already maxed out on IOPS just handling the workload. So moving the data from fast to slow takes weeks and impacts performance. The original intention was: [root at scg-gs0 ~]# cat policy_placement.txt RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%' RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' I would recommend that your total workload fits in both capacity and performance into your "primary" aka "fast" pool. I also just learned recently of the LIMIT parameter for placement, e.g. RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' LIMIT (95) RULE 'spillover' SET POOL 'sata0' So that would help with the pool not filling, but won't help with not having enough I/O for migration+clients. Regards, -- Alex Chekholko chekh at stanford.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Mon Jun 27 17:22:25 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Mon, 27 Jun 2016 17:22:25 +0100 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: <577152C1.8070802@pixitmedia.com> Hi Marc, Quite. We have had coverage of (1) and (2) in the API for some time. My overriding thought is that (QoS aside) if one executes an mmapplypolicy, for whatever reason, then a certain amount of resource is consumed. This may not be preferred. So let me rephrase to the users: in your real-world working environment, would you prefer: a) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc which is a sanity check, but not a policy run, hence no load b) mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. To be clear; you can currently achieve either launch with the API as it stands. However we'd like to know what the general concensus would prefer to be the norm in such an implementation. Regarding dry run functionality this can be achieved globally as follows: setDryRun(True) or as a more granular decorator: @dryrun ... def delete_filesets(): Jez On 20/06/16 16:03, Marc A Kaplan wrote: > *Jez,* > > Regarding your recent post. Do the mmchpolicy and mmapplypolicy > commands have sufficient functionality for your purposes? > Are you suggesting some improvements? If so, what? Please provide > examples and/or specific suggestions. > > WRT your numbered items: > > (1) `mmchpolicy fsname -I policy-rules-file test` does a complete > syntax check on policy-rules-file and some other sanity checking. > > (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I > test -L 0` is another way to validate policy-rules-file. Rules like > MIGRATE and LIST that are interpreted by mmapplypolicy will be > scrutinized. For example check that each named pool is defined either > within the file system or by an EXTERNAL POOL rules. > > (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 > [other options]` will do a dry run and show what actions would be > performed on each file. > > > --marc > > ----------- > > API calls for Policies have the ability to run in 'test' or 'run' > mode. (ref: man mmapplypolicy) > > How would you expect to use dry-run functionality on a policy? > We have our own opinion, but we'd like to hear yours. > > 1) Validate the policy and print the policy content to stdout/other. > No mmapplypolicy is performed. > 2) Validate the policy and enforce mmapplypolicy with -I test > 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol > --flag1 --flag2 --etc > 4) Other > > Best regards, > > Jez > -- > Jez Tucker > Head of Research & Development > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media www.pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 27 18:35:33 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 27 Jun 2016 13:35:33 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: <577152C1.8070802@pixitmedia.com> References: <57598963.3040706@pixitmedia.com> <577152C1.8070802@pixitmedia.com> Message-ID: What I always recommend is to make a directory of test cases and run mmapplypolicy /gpfs-path/test-cases -I test (or -I yes if you want to try operating on the test files) -L 2 (or -L 6 or something in between) -P policy-rules [other-options] Because as I think you're saying mmapplypolicy gpfs-fsname -I test ... Can consume a lot of compute and IO resource, just scanning the metadata. From: Jez Tucker To: gpfsug main discussion list Date: 06/27/2016 12:22 PM Subject: Re: [gpfsug-discuss] Dry-Run functionality ? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Marc, Quite. We have had coverage of (1) and (2) in the API for some time. My overriding thought is that (QoS aside) if one executes an mmapplypolicy, for whatever reason, then a certain amount of resource is consumed. This may not be preferred. So let me rephrase to the users: in your real-world working environment, would you prefer: a) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc which is a sanity check, but not a policy run, hence no load b) mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. To be clear; you can currently achieve either launch with the API as it stands. However we'd like to know what the general concensus would prefer to be the norm in such an implementation. Regarding dry run functionality this can be achieved globally as follows: setDryRun(True) or as a more granular decorator: @dryrun ... def delete_filesets(): Jez On 20/06/16 16:03, Marc A Kaplan wrote: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media www.pixitmedia.com This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Wed Jun 29 16:38:09 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Wed, 29 Jun 2016 17:38:09 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Message-ID: <5773EB61.2040203@ugent.be> Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth From Robert.Oesterlin at nuance.com Wed Jun 29 16:41:11 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 29 Jun 2016 15:41:11 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Message-ID: Nothing "official" from IBM, but I believe it's been delayed until later in July. Not sure if IBM would comment publically on it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Kenneth Waegeman Reply-To: gpfsug main discussion list Date: Wednesday, June 29, 2016 at 10:38 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fudqCYcU99iTF3_mD3_zr_XvAB9BGNRjbIka32m0Kdw&s=GIVzGr_ag_wl2INE5BTuO1fDQ0QOfKdo--ljaxj6FcI&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From raot at bnl.gov Wed Jun 29 16:43:25 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 11:43:25 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy Message-ID: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 29 16:46:57 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 29 Jun 2016 15:46:57 +0000 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: References: Message-ID: Hi Tejas, One thing that I have had occasional success with is a lazy unmount - i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does help in my experience, but it?s one thing I try before bouncing a node. HTHAL? Kevin On Jun 29, 2016, at 10:43 AM, Tejas Rao > wrote: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raot at bnl.gov Wed Jun 29 16:58:09 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 11:58:09 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: References: Message-ID: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> "umount -l" returns with "not mounted" message. See below. [root@ ~]# umount -l /gpfs01 umount: /gpfs01: not mounted [root@ ~]# umount -l /dev/gpfs01 umount: /dev/gpfs01: not mounted [root@ ~]# [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Thanks. On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: > Hi Tejas, > > One thing that I have had occasional success with is a lazy unmount - > i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does > help in my experience, but it?s one thing I try before bouncing a > node. HTHAL? > > Kevin > >> On Jun 29, 2016, at 10:43 AM, Tejas Rao > > wrote: >> >> Hi all, >> >> I cannot get the filesystem mounted again on few nodes. >> >> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >> mount: /dev/gpfs01 already mounted or /gpfs01 busy >> mmmount: Command failed. Examine previous error messages to determine >> cause. >> >> >> >> Things I have tried and checked. >> >> >> >> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >> /dev/gpfs01 and /gpfs01 are not seen anywhere. >> >> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >> See below. >> >> 3. mmumount gpfs01 -f >> >> >> A reboot of the client would probably fix it but as the client is >> used in production, this is not easy to do. >> >> >> Are there any tricks/hacks to remount the filesystem again without >> rebooting the client. >> >> >> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >> GPFS file systems >> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >> Shutting down! >> 'shutdown' command about to kill process 11431 >> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >> mmfsenv: Module mmfslinux is still in use. >> Unmount all GPFS file systems and issue the command: >> mmfsadm cleanup >> Unloading module mmfs26 >> ERROR: Module mmfs26 is in use >> mmfsenv: Error unloading module mmfs26. >> Unloading module mmfslinux >> ERROR: Module mmfslinux is in use by mmfs26 >> mmfsenv: Error unloading module mmfslinux. >> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >> [root@~]# >> >> Thanks. >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Jun 29 17:02:27 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Jun 2016 16:02:27 +0000 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> References: , <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> Message-ID: Process running which is either from gpfs or using it. We used to see it when we ran torque from gpfs and forgot to stop it before shutting down gpfs. Or if you have dsmrecalld (for example) running on the node. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Tejas Rao [raot at bnl.gov] Sent: 29 June 2016 16:58 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy "umount -l" returns with "not mounted" message. See below. [root@ ~]# umount -l /gpfs01 umount: /gpfs01: not mounted [root@ ~]# umount -l /dev/gpfs01 umount: /dev/gpfs01: not mounted [root@ ~]# [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Thanks. On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: Hi Tejas, One thing that I have had occasional success with is a lazy unmount - i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does help in my experience, but it?s one thing I try before bouncing a node. HTHAL? Kevin On Jun 29, 2016, at 10:43 AM, Tejas Rao <raot at bnl.gov> wrote: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From konstantin.arnold at unibas.ch Wed Jun 29 17:05:33 2016 From: konstantin.arnold at unibas.ch (Konstantin Arnold) Date: Wed, 29 Jun 2016 18:05:33 +0200 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> References: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> Message-ID: <5773F1CD.2000205@unibas.ch> Hi Tejas, we have observed this behavior too, typically when automounter or open file handles are involved. We tried to close the file handles (kill the process that has them open), next tried to unmount the automouter path and then mmumount filesystem again. In cases where we could not find and close the open file handles (e.g. process cannot be killed anymore) or you're stuck with unmounting the autoumounter path, we had to stop GPFS and verify the kernel module gets unloaded. In case the kernel module stays loaded, our only option was to reboot the node. Best Konstantin On 06/29/2016 05:58 PM, Tejas Rao wrote: > "umount -l" returns with "not mounted" message. See below. > > > [root@ ~]# umount -l /gpfs01 > umount: /gpfs01: not mounted > > [root@ ~]# umount -l /dev/gpfs01 > umount: /dev/gpfs01: not mounted > [root@ ~]# > > > [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 > Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... > mount: /dev/gpfs01 already mounted or /gpfs01 busy > mmmount: Command failed. Examine previous error messages to determine cause. > > Thanks. > > > On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: >> Hi Tejas, >> >> One thing that I have had occasional success with is a lazy unmount - >> i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does >> help in my experience, but it?s one thing I try before bouncing a >> node. HTHAL? >> >> Kevin >> >>> On Jun 29, 2016, at 10:43 AM, Tejas Rao >>> <raot at bnl.gov> wrote: >>> >>> Hi all, >>> >>> I cannot get the filesystem mounted again on few nodes. >>> >>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >>> mount: /dev/gpfs01 already mounted or /gpfs01 busy >>> mmmount: Command failed. Examine previous error messages to determine >>> cause. >>> >>> >>> >>> Things I have tried and checked. >>> >>> >>> >>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >>> /dev/gpfs01 and /gpfs01 are not seen anywhere. >>> >>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >>> See below. >>> >>> 3. mmumount gpfs01 -f >>> >>> >>> A reboot of the client would probably fix it but as the client is >>> used in production, this is not easy to do. >>> >>> >>> Are there any tricks/hacks to remount the filesystem again without >>> rebooting the client. >>> >>> >>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >>> GPFS file systems >>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >>> Shutting down! >>> 'shutdown' command about to kill process 11431 >>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >>> mmfsenv: Module mmfslinux is still in use. >>> Unmount all GPFS file systems and issue the command: >>> mmfsadm cleanup >>> Unloading module mmfs26 >>> ERROR: Module mmfs26 is in use >>> mmfsenv: Error unloading module mmfs26. >>> Unloading module mmfslinux >>> ERROR: Module mmfslinux is in use by mmfs26 >>> mmfsenv: Error unloading module mmfslinux. >>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >>> [root@~]# >>> >>> Thanks. >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- ------------------------------------------------------------------------------------------- Konstantin Arnold | University of Basel & SIB Klingelbergstrasse 50/70 | CH-4056 Basel | Phone: +41 61 267 15 82 Email: konstantin.arnold at unibas.ch From raot at bnl.gov Wed Jun 29 17:13:10 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 12:13:10 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <5773F1CD.2000205@unibas.ch> References: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> <5773F1CD.2000205@unibas.ch> Message-ID: Yes, open file handles were probably the issue, although I did not see them in lsof. I restarted my application and now I can mount /gpfs01/ again. Thanks, Tejas. On 6/29/2016 12:05, Konstantin Arnold wrote: > Hi Tejas, > > we have observed this behavior too, typically when automounter or open > file handles are involved. We tried to close the file handles (kill > the process that has them open), next tried to unmount the automouter > path and then mmumount filesystem again. In cases where we could not > find and close the open file handles (e.g. process cannot be killed > anymore) or you're stuck with unmounting the autoumounter path, we had > to stop GPFS and verify the kernel module gets unloaded. In case the > kernel module stays loaded, our only option was to reboot the node. > > Best > Konstantin > > > > On 06/29/2016 05:58 PM, Tejas Rao wrote: >> "umount -l" returns with "not mounted" message. See below. >> >> >> [root@ ~]# umount -l /gpfs01 >> umount: /gpfs01: not mounted >> >> [root@ ~]# umount -l /dev/gpfs01 >> umount: /dev/gpfs01: not mounted >> [root@ ~]# >> >> >> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >> Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... >> mount: /dev/gpfs01 already mounted or /gpfs01 busy >> mmmount: Command failed. Examine previous error messages to determine >> cause. >> >> Thanks. >> >> >> On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: >>> Hi Tejas, >>> >>> One thing that I have had occasional success with is a lazy unmount - >>> i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does >>> help in my experience, but it?s one thing I try before bouncing a >>> node. HTHAL? >>> >>> Kevin >>> >>>> On Jun 29, 2016, at 10:43 AM, Tejas Rao >>>> <raot at bnl.gov> wrote: >>>> >>>> Hi all, >>>> >>>> I cannot get the filesystem mounted again on few nodes. >>>> >>>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >>>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >>>> mount: /dev/gpfs01 already mounted or /gpfs01 busy >>>> mmmount: Command failed. Examine previous error messages to determine >>>> cause. >>>> >>>> >>>> >>>> Things I have tried and checked. >>>> >>>> >>>> >>>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >>>> /dev/gpfs01 and /gpfs01 are not seen anywhere. >>>> >>>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >>>> See below. >>>> >>>> 3. mmumount gpfs01 -f >>>> >>>> >>>> A reboot of the client would probably fix it but as the client is >>>> used in production, this is not easy to do. >>>> >>>> >>>> Are there any tricks/hacks to remount the filesystem again without >>>> rebooting the client. >>>> >>>> >>>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >>>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >>>> GPFS file systems >>>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >>>> Shutting down! >>>> 'shutdown' command about to kill process 11431 >>>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >>>> mmfsenv: Module mmfslinux is still in use. >>>> Unmount all GPFS file systems and issue the command: >>>> mmfsadm cleanup >>>> Unloading module mmfs26 >>>> ERROR: Module mmfs26 is in use >>>> mmfsenv: Error unloading module mmfs26. >>>> Unloading module mmfslinux >>>> ERROR: Module mmfslinux is in use by mmfs26 >>>> mmfsenv: Error unloading module mmfslinux. >>>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >>>> [root@~]# >>>> >>>> Thanks. >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> ? >>> Kevin Buterbaugh - Senior System Administrator >>> Vanderbilt University - Advanced Computing Center for Research and >>> Education >>> Kevin.Buterbaugh at vanderbilt.edu >>> - (615)875-9633 >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > From S.J.Thompson at bham.ac.uk Wed Jun 29 17:18:50 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Jun 2016 16:18:50 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: <5773EB61.2040203@ugent.be> References: <5773EB61.2040203@ugent.be> Message-ID: I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From daniel.kidger at uk.ibm.com Wed Jun 29 17:25:07 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 29 Jun 2016 16:25:07 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: Message-ID: Informally I hear that we a still a couple of weeks away. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 29 Jun 2016, 18:19:07, S.J.Thompson at bham.ac.uk wrote: From: S.J.Thompson at bham.ac.uk To: gpfsug-discuss at spectrumscale.org Cc: Date: 29 Jun 2016 18:19:07 Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discussUnless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Thu Jun 30 11:46:16 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Thu, 30 Jun 2016 12:46:16 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: References: <5773EB61.2040203@ugent.be> Message-ID: The GA date moved to July 29th. A gating issue was found late during the testing and nobody wanted to risk to release the product with this issue and provide enough time to make sure the issue is fixed and well tested. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect /PMP - Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-6131-84-3465 IBM Deutschland Mobile: +49-170-6346601 Hechtsheimer Str. 2 E-Mail: stschmid at de.ibm.com 55131 Mainz Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 29.06.2016 18:19 Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Thu Jun 30 11:56:04 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Thu, 30 Jun 2016 12:56:04 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: References: <5773EB61.2040203@ugent.be> Message-ID: <5774FAC4.5060705@ugent.be> Thanks for all the info, it's good to know! Kind regards, Kenneth On 30/06/16 12:46, Stefan Schmidt wrote: > The GA date moved to July 29th. A gating issue was found late during > the testing and nobody wanted to risk to release the product with this > issue and provide enough time to make sure the issue is fixed and well > tested. > > Mit freundlichen Gr??en / Kind regards > > *Stefan Schmidt* > > Scrum Master IBM Spectrum Scale GUI / Senior IT Architect /PMP - Dept. > M069 / IBM Spectrum Scale Software Development > > IBM Systems Group > > IBM Deutschland > > ------------------------------------------------------------------------ > > Phone: +49-6131-84-3465 IBM Deutschland > Mobile: +49-170-6346601 Hechtsheimer Str. 2 > E-Mail: stschmid at de.ibm.com 55131 Mainz > > > Germany > ------------------------------------------------------------------------ > > IBM Deutschland Research & Development GmbH / Vorsitzende des > Aufsichtsrats: Martina Koederitz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht > Stuttgart, HRB 243294 > > > > > > > > From: "Simon Thompson (Research Computing - IT Services)" > > To: gpfsug main discussion list > Date: 29.06.2016 18:19 > Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > > I'm not sure they announced... Someone may have said that they > expected June, you did read all the disclaimer in the slide decks right ;) > > But seriously, its not a formal product announcement platform, so > whilst people may talk about features, ideas etc... It shouldn't be > held as a commitment. > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth > Waegeman [kenneth.waegeman at ugent.be] > Sent: 29 June 2016 16:38 > To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 > > Hi all, > > On the User meeting in London a month ago, they announced that SS 4.2.1 > would be released somewhere in June. Since I did not found any release > notes or packages at the passport advantage portal, I guess it it is not > yet released? Does someone knows what would be the timing for this? > > Thank you! > > Cheers, > Kenneth > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From service at metamodul.com Thu Jun 30 12:53:11 2016 From: service at metamodul.com (- -) Date: Thu, 30 Jun 2016 13:53:11 +0200 (CEST) Subject: [gpfsug-discuss] Generic HA Environment Message-ID: <743599299.124585.fca32d27-f514-4edd-af56-0af407bd9929.open-xchange@email.1und1.de> Hi IBM folks, i would like to know if the current High Available (HA) infrastructure used by the Cluster Export Services (ces) like SMB & NFS can be used for private extensions meaning i am able to add additional service by my own. Thus there would be no need to write my own HA Infrastructre. tia Hajo From Mark.Bush at siriuscom.com Thu Jun 30 15:01:46 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Jun 2016 14:01:46 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store Message-ID: I know in the trial VM it recommends using CyberDuck to get to the object side of scale. Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush | Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions | www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Jun 30 15:29:17 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 30 Jun 2016 14:29:17 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: The object side is just openstack swift, so f.ex: https://github.com/openstack/python-swiftclient/blob/master/README.rst -jf tor. 30. jun. 2016 kl. 16.01 skrev Mark.Bush at siriuscom.com < Mark.Bush at siriuscom.com>: > I know in the trial VM it recommends using CyberDuck to get to the object > side of scale. Does anyone have a programmatic example of doing this? > Python would be preferred here. > > > > Mark R. Bush | Solutions Architect > Mobile: 210.237.8415 | mark.bush at siriuscom.com > Sirius Computer Solutions | www.siriuscom.com > 10100 Reunion Place, Suite 500, San Antonio, TX 78216 > > > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, and > exempt from disclosure under applicable law. If you are not the intended > recipient, you are hereby notified that any use, dissemination, > distribution, or copying of this communication is strictly prohibited. This > message may be viewed by parties at Sirius Computer Solutions other than > those named in the message header. This message does not contain an > official representation of Sirius Computer Solutions. If you have received > this communication in error, notify Sirius Computer Solutions immediately > and (i) destroy this message if a facsimile or (ii) delete this message > immediately if this is an electronic communication. Thank you. > Sirius Computer Solutions > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Thu Jun 30 15:33:31 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Thu, 30 Jun 2016 15:33:31 +0100 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: <57752DBB.3070403@pixitmedia.com> Hi Depending what your endpoint object store type is... https://github.com/openstack/python-swiftclient http://boto.cloudhackers.com/en/latest/ Though, I highly recommend: https://libcloud.apache.org/ I.E. http://libcloud.readthedocs.io/en/latest/storage/examples.html Best, Jez On 30/06/16 15:01, Mark.Bush at siriuscom.com wrote: > > I know in the trial VM it recommends using CyberDuck to get to the > object side of scale. Does anyone have a programmatic example of > doing this? Python would be preferred here. > > Mark R. Bush | Solutions Architect > Mobile: 210.237.8415 | mark.bush at siriuscom.com > Sirius Computer Solutions | www.siriuscom.com > 10100 Reunion Place, Suite 500, San Antonio, TX 78216 > > This message (including any attachments) is intended only for the use > of the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, > dissemination, distribution, or copying of this communication is > strictly prohibited. This message may be viewed by parties at Sirius > Computer Solutions other than those named in the message header. This > message does not contain an official representation of Sirius Computer > Solutions. If you have received this communication in error, notify > Sirius Computer Solutions immediately and (i) destroy this message if > a facsimile or (ii) delete this message immediately if this is an > electronic communication. Thank you. > > Sirius Computer Solutions > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media / ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Jun 30 15:48:00 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Jun 2016 14:48:00 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: <57752DBB.3070403@pixitmedia.com> References: <57752DBB.3070403@pixitmedia.com> Message-ID: <19124267-08D0-4232-BBF0-835653D8D4BB@siriuscom.com> Thanks to Jan-Frode and Jez. I will use the import swiftclient mechanism for this. From: Jez Tucker Organization: Pixit Media Limited Reply-To: gpfsug main discussion list Date: Thursday, June 30, 2016 at 9:33 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] restful way to get at SS OBJ store Hi Depending what your endpoint object store type is... https://github.com/openstack/python-swiftclient http://boto.cloudhackers.com/en/latest/ Though, I highly recommend: https://libcloud.apache.org/ I.E. http://libcloud.readthedocs.io/en/latest/storage/examples.html Best, Jez On 30/06/16 15:01, Mark.Bush at siriuscom.com wrote: I know in the trial VM it recommends using CyberDuck to get to the object side of scale. Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush | Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions | www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media / ArcaStream www.pixitmedia.com www.arcastream.com [http://pixitmedia.com/sig/sig-cio.jpg] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Thu Jun 30 16:37:05 2016 From: billowen at us.ibm.com (Bill Owen) Date: Thu, 30 Jun 2016 08:37:05 -0700 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: Also see: http://docs.openstack.org/developer/swift/api/object_api_v1_overview.html The trial VM has python-swiftclient installed. You can execute basic swift commands as follows: source openrc #define environment variables for account, user credentials etc. swift stat #show account statistics swift upload One helpful trick is to execute the client commands with --debug flag. This will show the exact http request that the client code is sending. For example: # swift --debug stat INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): client28 DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_7c0243a523cf436e9a0e79c9b7083f9e HTTP/1.1" 204 0 DEBUG:swiftclient:REQ: curl -i http://client28:8080/v1/AUTH_7c0243a523cf436e9a0e79c9b7083f9e -I -H "X-Auth-Token: " DEBUG:swiftclient:RESP STATUS: 204 No Content DEBUG:swiftclient:RESP HEADERS: [('content-length', '0'), ('x-account-object-count', '0'), ('x-account-project-domain-id', 'default'), ('x-account-storage-policy-policy-0-bytes-used', '0'), ('x-account-storage-policy-policy-0-container-count', '10'), ('x-timestamp', '1467238461.63001'), ('x-account-storage-policy-policy-0-object-count', '0'), ('x-trans-id', 'txd3e3d9ca87fe4e23a5c9c-0057753b57'), ('date', 'Thu, 30 Jun 2016 15:31:36 GMT'), ('x-account-bytes-used', '0'), ('x-account-container-count', '10'), ('content-type', 'text/plain; charset=utf-8'), ('accept-ranges', 'bytes')] Account: AUTH_7c0243a523cf436e9a0e79c9b7083f9e Containers: 10 Objects: 0 Bytes: 0 Containers in policy "policy-0": 10 Objects in policy "policy-0": 0 Bytes in policy "policy-0": 0 X-Account-Project-Domain-Id: default X-Timestamp: 1467238461.63001 X-Trans-Id: txd3e3d9ca87fe4e23a5c9c-0057753b57 Content-Type: text/plain; charset=utf-8 Accept-Ranges: bytes Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: Jan-Frode Myklebust To: "gpfsug-discuss at spectrumscale.org" Date: 06/30/2016 07:29 AM Subject: Re: [gpfsug-discuss] restful way to get at SS OBJ store Sent by: gpfsug-discuss-bounces at spectrumscale.org The object side is just openstack swift, so f.ex: https://github.com/openstack/python-swiftclient/blob/master/README.rst -jf tor. 30. jun. 2016 kl. 16.01 skrev Mark.Bush at siriuscom.com < Mark.Bush at siriuscom.com>: I know in the trial VM it recommends using CyberDuck to get to the object side of scale.? Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush?| Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions?|?www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Wed Jun 1 15:55:57 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 1 Jun 2016 14:55:57 +0000 Subject: [gpfsug-discuss] Building portability layer fails In-Reply-To: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk> References: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk>, <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com> Message-ID: <201606011456.u51EuIRY007236@d06av01.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 1 16:03:04 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 1 Jun 2016 15:03:04 +0000 Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All versions In-Reply-To: References: Message-ID: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Wed Jun 1 16:21:28 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Wed, 01 Jun 2016 16:21:28 +0100 Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All versions In-Reply-To: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> References: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> Message-ID: <1464794488.3773.15.camel@buzzard.phy.strath.ac.uk> On Wed, 2016-06-01 at 15:03 +0000, Daniel Kidger wrote: > By the way the files affected are all binaries so not as obvious a > security risk as allowing a backdoor to an interactive ksh shell as > root. Well yeah, Linux has not allowed setuid on scripts of any kind for around 20 years now. Hence even if you are running on AIX they will be binaries because otherwise you would have to maintain a script for AIX and a binary for Linux. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From douglasof at us.ibm.com Wed Jun 1 20:54:04 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Wed, 1 Jun 2016 15:54:04 -0400 Subject: [gpfsug-discuss] IBM Webinar Message-ID: <20160601195415.A2EA26E044@b03ledav001.gho.boulder.ibm.com> Not everyone can make it to a User Group, so we are kicking off a series of webinars with IBM Spectrum Scale experts. Join us for the first one - and bring questions. I will be hosting and having two of our architects on the call. Title: Building a Better Data Ocean: what's new in IBM Spectrum Scale Date: June 16, 12:00 EST Speakers: Piyush Chaudhary, Spectrum Scale Architect, Big Data & Analytics Rob Basham Transparent Cloud tiering and Cloud archive Storage lead architect URL: https://attendee.gotowebinar.com/register/6431264887578253057 -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Wed Jun 1 23:04:45 2016 From: billowen at us.ibm.com (Bill Owen) Date: Wed, 1 Jun 2016 15:04:45 -0700 Subject: [gpfsug-discuss] SECOND CHANCE - OpenStack & Spectrum Scale Usage Survey In-Reply-To: References: Message-ID: <20160601220453.5CD0EC604C@b03ledav006.gho.boulder.ibm.com> If you already filled out the OpenStack & Spectrum Scale Usage Survey, Thank You! If you haven't yet, there's still time to be heard - Please take 10 minutes to answer the questions in this short survey: https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey Thank you, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: Bill Owen/Tucson/IBM To: gpfsug-discuss at gpfsug.org Date: 05/14/2016 05:05 AM Subject: OpenStack & Spectrum Scale Usage Survey We in Spectrum Scale development are very interested in learning about your current & planned usage of OpenStack with Spectrum Scale, as well as plans for deploying containers in your Spectrum Scale environment. We would like to use this information to help shape our road map in this area over the next 18-24 months. Please take 10 minutes to answer the questions in this short survey: https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey Thank you, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Fri Jun 3 14:38:01 2016 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 3 Jun 2016 13:38:01 +0000 Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 on Haswell/Broadwell? Message-ID: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding "nosmap" to the kernel command line in grub prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.) >From https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html, section 3.2: Note: In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor * Disable the Supervisor Mode Access Prevention (smap) kernel parameter * Reboot the RHEL 7 node before using GPFS Some observations worth noting: 1. We've been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs? 2. It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without "nosmap". Instead, an error message describing the fix would have made my day. 3. I'm going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. Thanks, Paul Sanchez -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jun 7 19:25:39 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 7 Jun 2016 11:25:39 -0700 Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 onHaswell/Broadwell? In-Reply-To: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> References: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> Message-ID: Hi Paul, Yes, GPFS certainly needs to behave better in this situation. We are currently working on proper support for running on newer hardware that supports Superuser Mode Access Prevention (SMAP) instructions. I believe those are new to Broadwell CPUs, but there's some confusing info out there, I'm not positive what the deal is with Haswell. For the time being, booting with the "nosmap" kernel parameter is the workaround, but you're absolutely correct, the code needs to fail more gracefully when SMAP is enabled. We'll fix that. The current FAQ structure is, without question, suboptimal. We're looking for a better format to present this information, along the lines of more modern approaches like a structured Knowledge Base. The problem is recognized, on our end, but we've been having hard time making forward progress on this. yuri From: "Sanchez, Paul" To: "gpfsug main discussion list (gpfsug-discuss at spectrumscale.org)" , Date: 06/03/2016 06:38 AM Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 on Haswell/Broadwell? Sent by: gpfsug-discuss-bounces at spectrumscale.org After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding ?nosmap? to the kernel command line in grub prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.) From https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html, section 3.2: Note: In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor Disable the Supervisor Mode Access Prevention (smap) kernel parameter Reboot the RHEL 7 node before using GPFS Some observations worth noting: 1. We?ve been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs? 2. It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without ?nosmap?. Instead, an error message describing the fix would have made my day. 3. I?m going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. Thanks, Paul Sanchez _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 8 15:50:10 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 8 Jun 2016 14:50:10 +0000 Subject: [gpfsug-discuss] GPFS Tuning Wiki Page Message-ID: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> Hi All, https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Linux%20Tuning The above Wiki page describes itself as "GPFS network tuning parameters for large clusters on Linux.? How does GPFS multi-cluster play into this equation? For example, we?ve got a ~700 node traditional HPC cluster where these parameters would obviously come into play. But we?ve also got another 9 node GPFS cluster that is mainly accessed via CNFS and CIFS from labs around campus. However, it?s filesystem is also natively GPFS mounted on the HPC cluster via GPFS multi-cluster. Do I therefore need to apply the tuning referenced in the Wiki page above to the 9 node cluster? Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Jun 8 21:31:04 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 8 Jun 2016 22:31:04 +0200 Subject: [gpfsug-discuss] GPFS Tuning Wiki Page In-Reply-To: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> References: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> Message-ID: An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Thu Jun 9 16:21:07 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Thu, 9 Jun 2016 16:21:07 +0100 Subject: [gpfsug-discuss] Dry-Run functionality Message-ID: <57598963.3040706@pixitmedia.com> Hello all, I wonder if you could have your take on your ideal dry-run functionality. Post London User Group, we had much feedback regarding the need for dry-run functionality. Therefore we have implemented dry-run functionality for our GPFS/Scale Python API across all calls. API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development Pixit Media & ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Luke.Raimbach at crick.ac.uk Thu Jun 9 16:26:49 2016 From: Luke.Raimbach at crick.ac.uk (Luke Raimbach) Date: Thu, 9 Jun 2016 15:26:49 +0000 Subject: [gpfsug-discuss] Number of Inodes in chained AFM Caches Message-ID: Hi All, We have a situation where we are chaining AFM caches together: local update on the end of a read-only cache of a non GPFS NFS share. Essentially we have an intermediate read-only cache which can pull multiple data sources in quickly over the local network at the remote site, staging the data for WAN transfer into the ultimate destination which is a local update cache. In our first set of test runs, I am seeing a large discrepancy in the number of allocated inodes in the intermediate cache and the final local-update cache. We have sized the intermediate read-only caches to contain about 10% more "Maximum number of inodes" than the source NFS share (to accommodate any growth that happens while we transfer the data). We have also size the final local-update cache to have the same maximum number of inodes. The number of allocated inodes in the intermediate read-only cache shows the number of files in the source NFS share. However in the local-update shares, I always see 1500800 allocated and am concerned that the destination is not reading all the file-system metadata from the intermediate cache. Any ideas what could be going on here? Cheers, Luke. Luke Raimbach? Senior HPC Data and Storage Systems Engineer, The Francis Crick Institute, Gibbs Building, 215 Euston Road, London NW1 2BE. E: luke.raimbach at crick.ac.uk W: www.crick.ac.uk The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE. From Robert.Oesterlin at nuance.com Sun Jun 12 01:20:23 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Sun, 12 Jun 2016 00:20:23 +0000 Subject: [gpfsug-discuss] GPFSUG Meeting at Argonne 6/10 - Presentations Message-ID: <108228BB-1A66-4CE4-86D1-0CE9B3C8196F@nuance.com> The presentations from the user group meeting on June10th at Argronne labs are now available on the spectrumscale.org site here: http://www.spectrumscale.org/presentations/ Both my myself and Kristy Kallback-Rose thank the following who helped make it happen: Argonne Labs: Bill Allcock, Gordon McPheeters IBM team: Doris Conti, Doug O'flaherty, Ulf Troppens Along with all the presenters! We'll be following up with a more detailed report in the coming days, look for that on the spectrumscale.org site too. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid SSUG-USA Co-Principal -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Mon Jun 13 17:11:07 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 12:11:07 -0400 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> Message-ID: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From cphoffma at lanl.gov Mon Jun 13 17:19:02 2016 From: cphoffma at lanl.gov (Hoffman, Christopher P) Date: Mon, 13 Jun 2016 16:19:02 +0000 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca>, <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: Hi Jaime, What in particular would you like explained more? I'd be more than happy to discuss things further. Chris ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] Sent: Monday, June 13, 2016 10:11 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS on ZFS? I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From stijn.deweirdt at ugent.be Mon Jun 13 17:25:55 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Mon, 13 Jun 2016 18:25:55 +0200 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: <90312ed5-f263-a973-c768-28f535dc1316@ugent.be> hi chris, do you have any form of HA for the zfs blockdevices/jbod (eg when a nsd reboots/breaks/...)? or do you rely on replication within GPFS? stijn On 06/13/2016 06:19 PM, Hoffman, Christopher P wrote: > Hi Jaime, > > What in particular would you like explained more? I'd be more than happy to discuss things further. > > Chris > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] > Sent: Monday, June 13, 2016 10:11 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > > I just came across this presentation on "GPFS with underlying ZFS > block devices", by Christopher Hoffman, Los Alamos National Lab, > although some of the > implementation remains obscure. > > http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf > > It would be great to have more details, in particular the possibility > of straight use of GPFS on ZFS, instead of the 'archive' use case as > described on the presentation. > > Thanks > Jaime > > > > > Quoting "Jaime Pinto" : > >> Since we can not get GNR outside ESS/GSS appliances, is anybody using >> ZFS for software raid on commodity storage? >> >> Thanks >> Jaime >> >> > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From pinto at scinet.utoronto.ca Mon Jun 13 17:26:27 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 12:26:27 -0400 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca>, <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> Hi Chris As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. How is that accomplished when you use ZFS for software raid? On page 4 I see this info, and I'm trying to interpret it: General Configuration ... * zvols * nsddevices ? echo "zdX generic? Thanks Jaime Quoting "Hoffman, Christopher P" : > Hi Jaime, > > What in particular would you like explained more? I'd be more than > happy to discuss things further. > > Chris > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] > Sent: Monday, June 13, 2016 10:11 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > > I just came across this presentation on "GPFS with underlying ZFS > block devices", by Christopher Hoffman, Los Alamos National Lab, > although some of the > implementation remains obscure. > > http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf > > It would be great to have more details, in particular the possibility > of straight use of GPFS on ZFS, instead of the 'archive' use case as > described on the presentation. > > Thanks > Jaime > > > > > Quoting "Jaime Pinto" : > >> Since we can not get GNR outside ESS/GSS appliances, is anybody using >> ZFS for software raid on commodity storage? >> >> Thanks >> Jaime >> >> > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From bsallen at alcf.anl.gov Mon Jun 13 17:33:54 2016 From: bsallen at alcf.anl.gov (Allen, Benjamin S.) Date: Mon, 13 Jun 2016 16:33:54 +0000 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> Message-ID: <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Jaime, See https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm. An example I have for add /dev/nvme* devices: * GPFS doesn't know how that /dev/nvme* are valid block devices, use a user exit script to let it know about it cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices * Edit /var/mmfs/etc/nsddevices, and add to linux section: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) do echo $dev generic done fi * Copy edited nsddevices to the rest of the nodes at the same path for host in n01 n02 n03 n04; do scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices done Ben > On Jun 13, 2016, at 11:26 AM, Jaime Pinto wrote: > > Hi Chris > > As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. > > How is that accomplished when you use ZFS for software raid? > On page 4 I see this info, and I'm trying to interpret it: > > General Configuration > ... > * zvols > * nsddevices > - echo "zdX generic" > > > Thanks > Jaime > > Quoting "Hoffman, Christopher P" : > >> Hi Jaime, >> >> What in particular would you like explained more? I'd be more than happy to discuss things further. >> >> Chris >> ________________________________________ >> From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] >> Sent: Monday, June 13, 2016 10:11 >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >> >> I just came across this presentation on "GPFS with underlying ZFS >> block devices", by Christopher Hoffman, Los Alamos National Lab, >> although some of the >> implementation remains obscure. >> >> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >> >> It would be great to have more details, in particular the possibility >> of straight use of GPFS on ZFS, instead of the 'archive' use case as >> described on the presentation. >> >> Thanks >> Jaime >> >> >> >> >> Quoting "Jaime Pinto" : >> >>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>> ZFS for software raid on commodity storage? >>> >>> Thanks >>> Jaime >>> >>> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Mon Jun 13 18:53:41 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 13 Jun 2016 13:53:41 -0400 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: How do you set the size of a ZFS file that is simulating a GPFS disk? How do "tell" GPFS about that? How efficient is this layering, compared to just giving GPFS direct access to the same kind of LUNs that ZFS is using? Hmmm... to partially answer my question, I do something similar, but strictly for testing non-performance critical GPFS functions. On any file system one can: dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a fake 3GB disk for GPFS Then use a GPFS nsd configuration record like this: %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra servers=bog-xxx Which starts out as sparse and the filesystem will dynamically "grow" as GPFS writes to it... But I have no idea how well this will work for a critical "production" system... tx, marc kaplan. From: "Allen, Benjamin S." To: gpfsug main discussion list Date: 06/13/2016 12:34 PM Subject: Re: [gpfsug-discuss] GPFS on ZFS? Sent by: gpfsug-discuss-bounces at spectrumscale.org Jaime, See https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm . An example I have for add /dev/nvme* devices: * GPFS doesn't know how that /dev/nvme* are valid block devices, use a user exit script to let it know about it cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices * Edit /var/mmfs/etc/nsddevices, and add to linux section: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) do echo $dev generic done fi * Copy edited nsddevices to the rest of the nodes at the same path for host in n01 n02 n03 n04; do scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices done Ben > On Jun 13, 2016, at 11:26 AM, Jaime Pinto wrote: > > Hi Chris > > As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. > > How is that accomplished when you use ZFS for software raid? > On page 4 I see this info, and I'm trying to interpret it: > > General Configuration > ... > * zvols > * nsddevices > - echo "zdX generic" > > > Thanks > Jaime > > Quoting "Hoffman, Christopher P" : > >> Hi Jaime, >> >> What in particular would you like explained more? I'd be more than happy to discuss things further. >> >> Chris >> ________________________________________ >> From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] >> Sent: Monday, June 13, 2016 10:11 >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >> >> I just came across this presentation on "GPFS with underlying ZFS >> block devices", by Christopher Hoffman, Los Alamos National Lab, >> although some of the >> implementation remains obscure. >> >> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >> >> It would be great to have more details, in particular the possibility >> of straight use of GPFS on ZFS, instead of the 'archive' use case as >> described on the presentation. >> >> Thanks >> Jaime >> >> >> >> >> Quoting "Jaime Pinto" : >> >>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>> ZFS for software raid on commodity storage? >>> >>> Thanks >>> Jaime >>> >>> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Mon Jun 13 19:02:51 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 14:02:51 -0400 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> As Marc, I also have questions related to performance. Assuming we let ZFS take care of the underlying software raid, what would be the difference between GPFS and Lustre for instance, for the "parallel serving" at scale part of the file system. What would keep GPFS from performing or functioning just as well? Thanks Jaime Quoting "Marc A Kaplan" : > How do you set the size of a ZFS file that is simulating a GPFS disk? How > do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct access > to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > > tx, marc kaplan. > > > > From: "Allen, Benjamin S." > To: gpfsug main discussion list > Date: 06/13/2016 12:34 PM > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Jaime, > > See > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > . An example I have for add /dev/nvme* devices: > > * GPFS doesn't know how that /dev/nvme* are valid block devices, use a > user exit script to let it know about it > > cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices > > * Edit /var/mmfs/etc/nsddevices, and add to linux section: > > if [[ $osName = Linux ]] > then > : # Add function to discover disks in the Linux environment. > for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) > do > echo $dev generic > done > fi > > * Copy edited nsddevices to the rest of the nodes at the same path > for host in n01 n02 n03 n04; do > scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices > done > > > Ben > >> On Jun 13, 2016, at 11:26 AM, Jaime Pinto > wrote: >> >> Hi Chris >> >> As I understand, GPFS likes to 'see' the block devices, even on a > hardware raid solution such as DDN's. >> >> How is that accomplished when you use ZFS for software raid? >> On page 4, I see this info, and I'm trying to interpret it: >> >> General Configuration >> ... >> * zvols >> * nsddevices >> - echo "zdX generic" >> >> >> Thanks >> Jaime >> >> Quoting "Hoffman, Christopher P" : >> >>> Hi Jaime, >>> >>> What in particular would you like explained more? I'd be more than > happy to discuss things further. >>> >>> Chris >>> ________________________________________ >>> From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] >>> Sent: Monday, June 13, 2016 10:11 >>> To: gpfsug main discussion list >>> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >>> >>> I just came across this presentation on "GPFS with underlying ZFS >>> block devices", by Christopher Hoffman, Los Alamos National Lab, >>> although some of the >>> implementation remains obscure. >>> >>> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >>> >>> It would be great to have more details, in particular the possibility >>> of straight use of GPFS on ZFS, instead of the 'archive' use case as >>> described on the presentation. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> Quoting "Jaime Pinto" : >>> >>>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>>> ZFS for software raid on commodity storage? >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> http://www.scinethpc.ca/testimonials >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.org >>> University of Toronto >>> 256 McCaul Street, Room 235 >>> Toronto, ON, M5T1W5 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of > Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of > Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From cphoffma at lanl.gov Mon Jun 13 19:15:37 2016 From: cphoffma at lanl.gov (Hoffman, Christopher P) Date: Mon, 13 Jun 2016 18:15:37 +0000 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> , <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> Message-ID: To specify the size of the disks GPFS uses one can use zvols. Then one can turn on the zfs setting sync=always to perform safe writes since I'm using SATA cards there is no BBU. In our testing, turning sync=on creates a 20%-30% decrease in overall throughput on writes. I do not have numbers of this setup vs hardware RAID6. Thanks, Chris ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] Sent: Monday, June 13, 2016 12:02 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS on ZFS! ... ? As Marc, I also have questions related to performance. Assuming we let ZFS take care of the underlying software raid, what would be the difference between GPFS and Lustre for instance, for the "parallel serving" at scale part of the file system. What would keep GPFS from performing or functioning just as well? Thanks Jaime Quoting "Marc A Kaplan" : > How do you set the size of a ZFS file that is simulating a GPFS disk? How > do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct access > to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > > tx, marc kaplan. > > > > From: "Allen, Benjamin S." > To: gpfsug main discussion list > Date: 06/13/2016 12:34 PM > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Jaime, > > See > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > . An example I have for add /dev/nvme* devices: > > * GPFS doesn't know how that /dev/nvme* are valid block devices, use a > user exit script to let it know about it > > cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices > > * Edit /var/mmfs/etc/nsddevices, and add to linux section: > > if [[ $osName = Linux ]] > then > : # Add function to discover disks in the Linux environment. > for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) > do > echo $dev generic > done > fi > > * Copy edited nsddevices to the rest of the nodes at the same path > for host in n01 n02 n03 n04; do > scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices > done > > > Ben > >> On Jun 13, 2016, at 11:26 AM, Jaime Pinto > wrote: >> >> Hi Chris >> >> As I understand, GPFS likes to 'see' the block devices, even on a > hardware raid solution such as DDN's. >> >> How is that accomplished when you use ZFS for software raid? >> On page 4, I see this info, and I'm trying to interpret it: >> >> General Configuration >> ... >> * zvols >> * nsddevices >> - echo "zdX generic" >> >> >> Thanks >> Jaime >> >> Quoting "Hoffman, Christopher P" : >> >>> Hi Jaime, >>> >>> What in particular would you like explained more? I'd be more than > happy to discuss things further. >>> >>> Chris >>> ________________________________________ >>> From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] >>> Sent: Monday, June 13, 2016 10:11 >>> To: gpfsug main discussion list >>> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >>> >>> I just came across this presentation on "GPFS with underlying ZFS >>> block devices", by Christopher Hoffman, Los Alamos National Lab, >>> although some of the >>> implementation remains obscure. >>> >>> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >>> >>> It would be great to have more details, in particular the possibility >>> of straight use of GPFS on ZFS, instead of the 'archive' use case as >>> described on the presentation. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> Quoting "Jaime Pinto" : >>> >>>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>>> ZFS for software raid on commodity storage? >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> http://www.scinethpc.ca/testimonials >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.org >>> University of Toronto >>> 256 McCaul Street, Room 235 >>> Toronto, ON, M5T1W5 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of > Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of > Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jonathan at buzzard.me.uk Mon Jun 13 20:05:20 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Mon, 13 Jun 2016 20:05:20 +0100 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: <575F03F0.2020605@buzzard.me.uk> On 13/06/16 18:53, Marc A Kaplan wrote: > How do you set the size of a ZFS file that is simulating a GPFS disk? > How do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct > access to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > For "testing" purposes I just create a logical volume and map it through to my bunch of GPFS KVM instances as a disk. Works a treat and SSD's are silly money these days so for testing performance is just fine. There was a 960GB SanDisk on offer for 160GBP last month. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From laurence at qsplace.co.uk Mon Jun 13 20:12:03 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Mon, 13 Jun 2016 22:12:03 +0300 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <575F03F0.2020605@buzzard.me.uk> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> <575F03F0.2020605@buzzard.me.uk> Message-ID: <9963179A-1D12-44BD-ABE7-5A607D0644D8@qsplace.co.uk> @JAB Same here passing the same LVM LV's through to multiple KVM instances works a treat for testing. -- Lauz On 13 June 2016 22:05:20 EEST, Jonathan Buzzard wrote: >On 13/06/16 18:53, Marc A Kaplan wrote: >> How do you set the size of a ZFS file that is simulating a GPFS disk? >> How do "tell" GPFS about that? >> >> How efficient is this layering, compared to just giving GPFS direct >> access to the same kind of LUNs that ZFS is using? >> >> Hmmm... to partially answer my question, I do something similar, but >> strictly for testing non-performance critical GPFS functions. >> On any file system one can: >> >> dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create >a >> fake 3GB disk for GPFS >> >> Then use a GPFS nsd configuration record like this: >> >> %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra >> servers=bog-xxx >> >> Which starts out as sparse and the filesystem will dynamically "grow" >as >> GPFS writes to it... >> >> But I have no idea how well this will work for a critical >"production" >> system... >> > >For "testing" purposes I just create a logical volume and map it >through >to my bunch of GPFS KVM instances as a disk. Works a treat and SSD's >are >silly money these days so for testing performance is just fine. There >was a 960GB SanDisk on offer for 160GBP last month. > >JAB. > >-- >Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk >Fife, United Kingdom. >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 14 21:50:08 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 14 Jun 2016 20:50:08 +0000 Subject: [gpfsug-discuss] QoS question Message-ID: Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jun 14 22:05:53 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 14 Jun 2016 14:05:53 -0700 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: GPFS proper (as opposed to GNR) isn't particularly picky about block devices. Any block device that GPFS can see, with help from an nsddevices user exit if necessary, is fair game, for those willing to blaze new trails. This applies to "real" devices, e.g. disk partitions or hardware RAID LUNs, and "virtual" ones, like software RAID devices. The device has to be capable to accepting IO requests of GPFS block size, but aside from that, Linux kernel does a pretty good job abstracting the realities of low-level implementation from the higher-level block device API. The basic problem with software RAID approaches is the lack of efficient HA. Since a given device is only visible to one node, if a node goes down, it takes the NSDs with it (as opposed to the more traditional twin-tailed disk model, when another NSD server can take over). So one would have to rely on GPFS data/metadata replication to get HA, and that is costly, in terms of disk utilization efficiency and data write cost. This is still an attractive model for some use cases, but it's not quite a one-to-one replacement for something like GNR for general use. yuri From: "Jaime Pinto" To: "gpfsug main discussion list" , Date: 06/13/2016 09:11 AM Subject: Re: [gpfsug-discuss] GPFS on ZFS? Sent by: gpfsug-discuss-bounces at spectrumscale.org I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Tue Jun 14 23:07:36 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 14 Jun 2016 18:07:36 -0400 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: Yes, in QOS for 4.2.0 there are some simple assumptions that may not make a lot of sense in some configurations, especially configurations with many (100s) of nodes mounting the same file system... You can try out what you suggested and in 4.2.0 I think it will pretty much work as you suggest -- essentially you are allocating 466 maintenance iops to every node, knowing that most of those nodes will not be using their allocation of IOPS. In later releases, you may find that we will address some of these kinds of quirks in QOS. QOS is a new feature for GPFS, and I don't think you'll find anything like it in any commercial file system offering. (Correct me with example(s) if I am wrong on this point.) So think of it as "release 1.0" (of QOS) and let us know how well it works for you and how it might be improved.... --marc of Spectrum(GP)FS From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2016 04:50 PM Subject: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Jun 15 06:54:54 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 15 Jun 2016 05:54:54 +0000 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: XFS on Irix had a feature similar to QoS, called GRIO (guaranteed rate I/O), where applications could reserve a given bandwidth. http://www.sgistuff.net/software/irixintro/documents/xfs-whitepaper.html Sounds somewhat similar to QoS, but focused on giving applications guaranteed bandwidth, not iops. -jf ons. 15. jun. 2016 kl. 00.08 skrev Marc A Kaplan : > Yes, in QOS for 4.2.0 there are some simple assumptions that may not make > a lot of sense in some configurations, especially configurations with many > (100s) of nodes mounting the same file system... You can try out what > you suggested and in 4.2.0 I think it will pretty much work as you suggest > -- essentially you are allocating 466 maintenance iops to every node, > knowing that most of those nodes will not be using their allocation of IOPS. > > In later releases, you may find that we will address some of these kinds > of quirks in QOS. > > QOS is a new feature for GPFS, and I don't think you'll find anything like > it in any commercial file system offering. (Correct me with example(s) if > I am wrong on this point.) > So think of it as "release 1.0" (of QOS) and let us know how well it works > for you and how it might be improved.... > > --marc of Spectrum(GP)FS > > > > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 06/14/2016 04:50 PM > Subject: [gpfsug-discuss] QoS question > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi All, > > We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to > dive into my first attempts at using the new QoS features. I want to make > sure I am understanding the documentation: > > "The IOPS values that you set in an mmchqos command apply to all I/O > operations that are issued by all the nodes that have the specified file > system mounted. You should adjust your allocations of IOPS accordingly. > > For example, if you 600 IOPS to the maintenance class, and there are six > nodes that have the file system mounted, then QoS allocates 100 IOPS to the > maintenance class of each node. If you then run maintenance commands that > affect only three of the nodes, the commands runs with an actual allocation > of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect > three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate > 1200 IOPS to the maintenanceclass. " > > We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret > the above assuming that I have determined that I want to allow 7,000 IOPs ? > please correct me if I?m wrong... > > 7,000 IOPs / 700 nodes would be 10 IOPs per node. > > But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that > are going to be doing the maintenance (a restripe, for example), so 7,000 / > 15 = 466.67. > > 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 > IOPs to the maintenance class? > > Thanks in advance? > > Kevin > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > > *Kevin.Buterbaugh at vanderbilt.edu* - > (615)875-9633 > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Wed Jun 15 12:03:10 2016 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 15 Jun 2016 13:03:10 +0200 Subject: [gpfsug-discuss] ISC U/G Agenda - Next Monday In-Reply-To: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com> References: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com> Message-ID: <201606151103.u5FAwsYp073979@mx0a-001b2d01.pphosted.com> Hello Everybody, here is a reminder for the User Group meeting at ISC in Frankfurt next Monday. Organized by IBM ... but same spirit as the meetings in London, New York and Chicago. See here for updated agenda and registration: http://www-05.ibm.com/de/events/isc16/ Looking forward to see many of you next Monday. Best regards, Ulf. -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Douglas O'flaherty" To: gpfsug-discuss at spectrumscale.org Date: 13.05.2016 17:02 Subject: [gpfsug-discuss] ISC U/G Agenda Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings: IBM is happy to announce the agenda for the joint IBM Spectrum Scale and Platform Computing User Group Agenda at ISC. We will finish on time to attend the opening reception. As with other user group meetings, the agenda includes user stories, updates on IBM Spectrum Scale & Platform LSF, and access to IBM experts and your peers. Please join us! To attend, please email Fabian.Beckmann at de.ibm.com so we can have an accurate count of attendees. Monday June 20, 2016 - 14:30-18:00 - Conference Room Konstant 14:30-14:40 [10 min] Welcome (Douglas o'Flaherty, IBM) 14:40-15:00 [20 min] Ten Reasons to Upgrade from GPFS 3.4 to Spectrum Scale 4.2 (Olaf Weiser, IBM) 15:00-15:30 [30 min] Shared Storage with in-memory latency: EMC DSSD D5 and IBM Spectrum Scale (Stefan Radtke, EMC) 15:30-16:00 [30 min] Workload scheduling and data management in a private cloud (Uwe Sommer, Airbus) 16:00-16:30 [30 min] Spectrum Scale site report (To be confirmed by customer) 16:30-17:00 [30 min] What's new in Platform LSF 10.1 & storage integration (Bill McMillan, IBM) 17:00-17:30 [30 min] What's new in Spectrum Scale 4.2.1 (Mathias Dietz, IBM) 17:30-18:00 [30 min] CORAL enhancements for Spectrum Scale (Sven Oehme, IBM) Looking forward to seeing you there! doug PS: IBMers can register their clients at this IBM link: https://w3-9033.ibm.com/events/ast/schedule/16isc.nsf _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Wed Jun 15 19:24:07 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Jun 2016 14:24:07 -0400 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: Thanks for that reference. Further research indicates that GRIO on XFS required Irix/SGI hardware features and GRIO is not in any recent implementations of XFS. Functionally GRIO, when it worked was kind of a logical complement or opposite of GPFS/QOS. GRIO was to be used to reserve bandwidth for critical applications; whereas GPFS 4.2/QOS primary purpose to restrict the IO bandwidth consumed by not-performance critical applications. Of course if you restrict set X to use no more than B, than you have effectively reserved TotalAvailable-B for the set ~X. It may interest you to know that the primary GPFS/QOS enforcement mechanisms are based on an implementation of Token Bucket ( wikipedia.org/wiki/Token_bucket , US Patent 9178827 ) --marc From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 06/15/2016 01:55 AM Subject: Re: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org XFS on Irix had a feature similar to QoS, called GRIO (guaranteed rate I/O), where applications could reserve a given bandwidth. http://www.sgistuff.net/software/irixintro/documents/xfs-whitepaper.html Sounds somewhat similar to QoS, but focused on giving applications guaranteed bandwidth, not iops. -jf ons. 15. jun. 2016 kl. 00.08 skrev Marc A Kaplan : Yes, in QOS for 4.2.0 there are some simple assumptions that may not make a lot of sense in some configurations, especially configurations with many (100s) of nodes mounting the same file system... You can try out what you suggested and in 4.2.0 I think it will pretty much work as you suggest -- essentially you are allocating 466 maintenance iops to every node, knowing that most of those nodes will not be using their allocation of IOPS. In later releases, you may find that we will address some of these kinds of quirks in QOS. QOS is a new feature for GPFS, and I don't think you'll find anything like it in any commercial file system offering. (Correct me with example(s) if I am wrong on this point.) So think of it as "release 1.0" (of QOS) and let us know how well it works for you and how it might be improved.... --marc of Spectrum(GP)FS From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2016 04:50 PM Subject: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 15:29:20 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 14:29:20 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? Message-ID: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 15:35:10 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 14:35:10 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: Message-ID: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> Hi All, And, of course, within 5 minutes of sending that e-mail I found the ?mmlsattr? command, which does what I need and which didn?t show up in my Google search results. I apologize for the noise? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamiedavis at us.ibm.com Fri Jun 17 16:04:17 2016 From: jamiedavis at us.ibm.com (James Davis) Date: Fri, 17 Jun 2016 15:04:17 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> References: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu>, Message-ID: An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 16:18:48 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 15:18:48 +0000 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: References: Message-ID: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Hi yet again all, Well, this has turned out to be an enlightening and surprising morning in GPFS land? What prompted my question below is this ? I am looking to use the new QoS features in GPFS 4.2. I have QoS enabled and am trying to get a baseline of IOPs so that I can determine how much I want to assign to the maintenance class (currently both maintenance and other are set to unlimited). To do this, I fired off a bonnie++ test from each of my NSD servers. The filesystem in question has two storage pools, the system pool and the capacity pool. The system pool is comprised of a couple of metadata only disks (SSD-based RAID 1 mirrors) and several data only disks (spinning HD-based RAID 6), while the capacity pool is comprised exclusively of data only disks (RAID 6). When the bonnie++?s were creating, reading, and rewriting the big file they create I was quite surprised to see mmlsqos show higher IOP?s on the capacity pool than the system pool by a factor of 10! As I was expecting those files to be being written to the system pool, this was quite surprising to me. Once I found the mmlsattr command, I ran it on one of the files being created and saw that it was indeed assigned to the capacity pool. The bonnie++?s finished before I could check the other files. I don?t have any file placement policies in effect for this filesystem, only file migration policies (each weekend any files in the system pool with an atime > 60 days get moved to the capacity pool and any files in the capacity pool with an atime < 60 days get moved to the system pool). In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.? This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don?t remember for sure) later. I was under the obviously mistaken impression that the ?first? data storage pool was the system pool, but that is clearly not correct. So my first question is, what is the definition of ?the first storage pool?? and my second question is, can the documentation be updated with the answer to my first question since it?s clearly ambiguous as written now? Thanks? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 16:21:48 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 15:21:48 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> Message-ID: <9C2AC549-B85E-40EE-92CA-98983A4B3D85@vanderbilt.edu> Hi Jamie, The much longer e-mail I just sent to the list will explain the background behind all this, but to answer your question I actually only needed to know which storage pool the file resided in. Obviously, if I know which NSDs it resides on that would tell me the storage pool. Thanks? Kevin On Jun 17, 2016, at 10:04 AM, James Davis > wrote: Hi Kevin, It looks like you can use mmlsattr to query the storage pool the file resides on. Is that all you needed to know, or did you need the specific disks on which it resides? If you're looking for the specific disks, how are you using the mmlsattr command to accomplish that? Jamie Jamie Davis GPFS Functional Verification Test (FVT) jamiedavis at us.ibm.com ----- Original message ----- From: "Buterbaugh, Kevin L" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: Re: [gpfsug-discuss] Can you query which NSDs a given file resides on? Date: Fri, Jun 17, 2016 10:36 AM Hi All, And, of course, within 5 minutes of sending that e-mail I found the ?mmlsattr? command, which does what I need and which didn?t show up in my Google search results. I apologize for the noise? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 17 16:22:44 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 11:22:44 -0400 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: Message-ID: Hint: tsdbfs More hints: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014215697 If you mess up, you didn't hear this from me ;-) From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/17/2016 10:30 AM Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamiedavis at us.ibm.com Fri Jun 17 16:50:26 2016 From: jamiedavis at us.ibm.com (James Davis) Date: Fri, 17 Jun 2016 15:50:26 +0000 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu>, Message-ID: An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Fri Jun 17 16:55:21 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 17 Jun 2016 16:55:21 +0100 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: <1466178921.3773.107.camel@buzzard.phy.strath.ac.uk> On Fri, 2016-06-17 at 15:18 +0000, Buterbaugh, Kevin L wrote: > Hi yet again all, > > > Well, this has turned out to be an enlightening and surprising morning > in GPFS land? > > > What prompted my question below is this ? I am looking to use the new > QoS features in GPFS 4.2. I have QoS enabled and am trying to get a > baseline of IOPs so that I can determine how much I want to assign to > the maintenance class (currently both maintenance and other are set to > unlimited). To do this, I fired off a bonnie++ test from each of my > NSD servers. > > > The filesystem in question has two storage pools, the system pool and > the capacity pool. The system pool is comprised of a couple of > metadata only disks (SSD-based RAID 1 mirrors) and several data only > disks (spinning HD-based RAID 6), while the capacity pool is comprised > exclusively of data only disks (RAID 6). > My preference in these scenarios is to have the system pool for metadata only. Then have a "fast" pool and a "slow" pool. Pool names more than 7 characters is a recipe for backup headaches with TSM at least, so "fast" and "slow" are deliberately chosen as names :) That way you can tweak block sizes more optimally. > > When the bonnie++?s were creating, reading, and rewriting the big file > they create I was quite surprised to see mmlsqos show higher IOP?s on > the capacity pool than the system pool by a factor of 10! As I was > expecting those files to be being written to the system pool, this was > quite surprising to me. Once I found the mmlsattr command, I ran it > on one of the files being created and saw that it was indeed assigned > to the capacity pool. The bonnie++?s finished before I could check > the other files. > > > I don?t have any file placement policies in effect for this > filesystem, only file migration policies (each weekend any files in > the system pool with an atime > 60 days get moved to the capacity pool > and any files in the capacity pool with an atime < 60 days get moved > to the system pool). Note due to the way applications handle saving files by creating a new file, moving files from slow to fast disk is mostly a waste of effort in my experience. By the time you do that, 99% of the time the user has finished with the file anyway. When they haven't generally the slow disk has gazillions of free IOPs anyway. > > In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS > file system does not have a placement policy installed, all the data > is stored in the first data storage pool.? > My understanding is that no placement policy and it can go anywhere, where anywhere is the next available NSD with free space. I found that you needed to do something like this at the end of your policy file for the default placement. /* move files over 1GB onto nearline storage and keep them there */ RULE 'big' MIGRATE FROM POOL 'system' TO POOL 'nearline' WHERE FILE_SIZE > 1073741824 /* migrate old files to the RAID6 disks to keep the RAID1 disks free */ RULE 'ilm' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(weighting) TO POOL 'slow' /* by default new files to the fast RAID1 disk unless full, then to RAID6 */ RULE 'new' SET POOL 'fast' LIMIT(95) RULE 'spillover' SET POOL 'slow' You have to stop allocation to the fast disk from ever approaching 100%. Otherwise a you get disk full errors on new file writes. Imagine you create a new file when there is 1KB of space free in the fast pool, then try and write 10KB of data; it will fail with an error. So first rule forces large files to the slow pool anyway (you can tweak this size to suite your environment) but it must be significantly less than the the space free when you hit 95% full on the fast disk. You then need to have a rule to kick the tiering migration in lower than when you start allocating to the slow pool. Finally you allocation new files to the fast disk till it gets quite full, before spilling over to the slow pool (which one presumes has loads of free capacity) if the fast pool fills up. Not shown you ideally need to kick in tiering migration every night so that it does not run during the day when users might notice the performance degradation. Ideally you want to have enough free space come the morning in your fast disk pool that all new files can be created there during the day. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From makaplan at us.ibm.com Fri Jun 17 16:56:59 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 11:56:59 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (doh!) In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: Thanks for asking! Prior to Release 4.2 system pool was the default pool for storing file data and you had to write at least one policy SET POOL rule to make use of any NSDs that you assigned to some other pool. Now if a file system is formatted or upgraded to level 4.2 or higher, and there is a data pool defined, and there are no SET POOL policy rules, then the "first" such data pool becomes the default storage pool for file data. "metadata" is always stored in system pool. So how is the "first" data pool determined? It's usually the first data pool added to the file system. For many customers and installations there is only one such pool, so no problem - and this is exactly what they wanted all along (since we introduced "pools" in release 3.1) We received complaints over the years along the lines: "Heh! Why the heck do you think I made a data pool? I don't want to know about silly SET POOL rules and yet another mm command (mmchpolicy)! Just do it!" Well in 4.2 we did. If you look into the /var/adm/ras/mmfs.log.* file(s) you will see that during mmmount 4.2 will tell you.... Fri Jun 17 09:31:26.637 2016: [I] Command: mount yy Fri Jun 17 09:31:27.625 2016: [I] Loaded policy 'for file system yy': Parsed 4 policy rules. Fri Jun 17 09:31:27.626 2016: Policy has no storage pool placement rules. Fri Jun 17 09:31:27.627 2016: [I] Data will be stored in pool 'xtra'. Notice we DID NOT change the behavior for file systems at level 4.1 or prior, even when you upgrade the software. But when you upgrade the file system to 4.2 (for example to use QOS) ... From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/17/2016 11:19 AM Subject: [gpfsug-discuss] Initial file placement Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi yet again all, Well, this has turned out to be an enlightening and surprising morning in GPFS land? What prompted my question below is this ? I am looking to use the new QoS features in GPFS 4.2. I have QoS enabled and am trying to get a baseline of IOPs so that I can determine how much I want to assign to the maintenance class (currently both maintenance and other are set to unlimited). To do this, I fired off a bonnie++ test from each of my NSD servers. The filesystem in question has two storage pools, the system pool and the capacity pool. The system pool is comprised of a couple of metadata only disks (SSD-based RAID 1 mirrors) and several data only disks (spinning HD-based RAID 6), while the capacity pool is comprised exclusively of data only disks (RAID 6). When the bonnie++?s were creating, reading, and rewriting the big file they create I was quite surprised to see mmlsqos show higher IOP?s on the capacity pool than the system pool by a factor of 10! As I was expecting those files to be being written to the system pool, this was quite surprising to me. Once I found the mmlsattr command, I ran it on one of the files being created and saw that it was indeed assigned to the capacity pool. The bonnie++?s finished before I could check the other files. I don?t have any file placement policies in effect for this filesystem, only file migration policies (each weekend any files in the system pool with an atime > 60 days get moved to the capacity pool and any files in the capacity pool with an atime < 60 days get moved to the system pool). In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.? This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don?t remember for sure) later. I was under the obviously mistaken impression that the ?first? data storage pool was the system pool, but that is clearly not correct. So my first question is, what is the definition of ?the first storage pool?? and my second question is, can the documentation be updated with the answer to my first question since it?s clearly ambiguous as written now? Thanks? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L < Kevin.Buterbaugh at Vanderbilt.Edu> wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 17 17:10:55 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 12:10:55 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (doh!) (DOH!) 4.1.1 In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: OOPS... The story I gave was almost correct. The cut over was at 4.1.1 (after 4.1 before 4.2) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Jun 17 17:20:22 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 12:20:22 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (DOC!) In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: (Doc?... I ain't about to read a manual! But just to cover our bases we did stick it into the 4.1.1 pubs) Policy improvements: This release includes the following policy improvements: ... (summary of changes, page xv) ... Implicit SET POOL 'first-data-pool' rule For file systems that are at or have been upgraded to 4.1.1, the system recognizes that, even if no policy rules have been installed to a file system by mmchpolicy , data files should be stored in a non-system pool if available (rather than in the system pool, which is the default for earlier releases). For more information, see the following: Information Lifecycle Management chapter in the IBM Spectrum Scale: Advanced Administration Guide ... ( mmchpolicy man page 298) ... For file systems that are at or have been upgraded to 4.1.1 or later: If there are no SET POOL policy rules installed to a file system by mmchpolicy, the system acts as if the single rule SET POOL 'first-data-pool' is in effect, where first-data-pool is the firstmost non-system pool that is available for file data storage, if such a non-system pool is available. (?Firstmost? is the first according to an internal index of all pools.) However, if there are no policy rules installed and there is no non-system pool, the system acts as if SET POOL 'system' is in effect. This change applies only to file systems that were created at or upgraded to 4.1.1. Until a file system is upgraded, if no SET POOL rules are present (set by mmchpolicy) for the file system, all data will be stored in the 'system' pool. For information on GPFS policies, see the IBM Spectrum Scale: Advanced Administration Guide. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 17:33:39 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 16:33:39 +0000 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (DOC!) In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: <78BBA4B7-505F-4D21-BA14-BC430D93F993@vanderbilt.edu> Hi Marc, Read every last one of the thousands of pages of documentation that comprises the GPFS documentation set before doing any GPFS upgrades ? yes, I apparently *do* need to do that! ;-) The paragraph that I find ambiguous can be found on page 20 of the GPFS 4.2 Advanced Administration Guide (3rd paragraph from bottom): The placement policy defining the initial placement of newly created files and the rules for placement of * | restored data must be installed into GPFS with the mmchpolicy command. If a GPFS file system does not * | have a placement policy installed, all the data is stored in the first data storage pool. Only one placement policy can be installed at a time. If you switch from one placement policy to another, or make changes to a placement policy, that action has no effect on existing files. However, newly created files are always placed according to the currently installed placement policy. In my opinion, it should at least contain a pointer the pages you reference below. Preferentially, it would contain the definition of ?first data storage pool?. Thanks? Kevin On Jun 17, 2016, at 11:20 AM, Marc A Kaplan > wrote: (Doc?... I ain't about to read a manual! But just to cover our bases we did stick it into the 4.1.1 pubs) Policy improvements: This release includes the following policy improvements: ... (summary of changes, page xv) ... Implicit SET POOL 'first-data-pool' rule For file systems that are at or have been upgraded to 4.1.1, the system recognizes that, even if no policy rules have been installed to a file system by mmchpolicy, data files should be stored in a non-system pool if available (rather than in the system pool, which is the default for earlier releases). For more information, see the following: Information Lifecycle Management chapter in the IBM Spectrum Scale: Advanced Administration Guide ... ( mmchpolicy man page 298) ... For file systems that are at or have been upgraded to 4.1.1 or later: If there are no SET POOL policy rules installed to a file system by mmchpolicy, the system acts as if the single rule SET POOL 'first-data-pool' is in effect, where first-data-pool is the firstmost non-system pool that is available for file data storage, if such a non-system pool is available. (?Firstmost? is the first according to an internal index of all pools.) However, if there are no policy rules installed and there is no non-system pool, the system acts as if SET POOL 'system' is in effect. This change applies only to file systems that were created at or upgraded to 4.1.1. Until a file system is upgraded, if no SET POOL rules are present (set by mmchpolicy) for the file system, all data will be stored in the 'system' pool. For information on GPFS policies, see the IBM Spectrum Scale: Advanced Administration Guide. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 14:45:34 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 13:45:34 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions Message-ID: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don't know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set "force user = root" all the snapshots are available. I think the answer is SAMBA, right? We're running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 14:49:36 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 13:49:36 +0000 Subject: [gpfsug-discuss] Dry-Run functionality In-Reply-To: <57598963.3040706@pixitmedia.com> References: <57598963.3040706@pixitmedia.com> Message-ID: IMO I?d want to see an explanation of what will happen in plain English. ?This will create a policy that moves any file bigger than 10MB and older than 30 days to storage pool XYZ?. You might be able to tell I?ve never used GPFS policies except for initial placement so the above example might not be possible anyway ? I?m very used to the Powershell switch ?-whatif? which basically said ?I am going to do the following:? Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jez Tucker Sent: 09 June 2016 16:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] Dry-Run functionality Hello all, I wonder if you could have your take on your ideal dry-run functionality. Post London User Group, we had much feedback regarding the need for dry-run functionality. Therefore we have implemented dry-run functionality for our GPFS/Scale Python API across all calls. API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development Pixit Media & ArcaStream www.pixitmedia.com www.arcastream.com [Image removed by sender.] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD000.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD000.jpg URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Jun 20 15:56:27 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 20 Jun 2016 14:56:27 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions In-Reply-To: References: Message-ID: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> Hi Richard, I can?t answer your question but I can tell you that we have experienced either the exact same thing you are or something very similar. It occurred for us after upgrading from GPFS 3.5 to 4.1.0.8 and it persists even after upgraded to GPFS 4.2.0.3 and the very latest sernet-samba. And to be clear, when we upgraded from GPFS 3.5 to 4.1 we did *not* upgrade SAMBA versions at that time. Therefore, I believe that something changed in GPFS. That doesn?t mean it?s GPFS? fault, of course. SAMBA may have been relying on a bugundocumented feature in GPFS that IBM fixed for all I know, and I?m obviously speculating here. The problem we see is that the .snapshots directory in each folder can be cd?d to but is empty. The snapshots are all there, however, if you: cd //.snapshots//rest/of/path/to/folder/in/question This obviously prevents users from being able to do their own recovery of files unless you do something like what you describe, which we are unwilling to do for security reasons. We have a ticket open with DDN? Kevin On Jun 20, 2016, at 8:45 AM, Sobey, Richard A > wrote: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don?t know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set ?force user = root? all the snapshots are available. I think the answer is SAMBA, right? We?re running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 16:03:28 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 15:03:28 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions In-Reply-To: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> References: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> Message-ID: Thanks Kevin. We are upgrading to GPFS 4.2 and CES in a few weeks but our customers have come to like previous versions and indeed it is sort of a selling point for us. Samba is the only thing we?ve changed recently after the badlock debacle so I?m tempted to blame that, but who knows. If (when) I find out I?ll let everyone know. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: 20 June 2016 15:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Snapshots / Windows previous versions Hi Richard, I can?t answer your question but I can tell you that we have experienced either the exact same thing you are or something very similar. It occurred for us after upgrading from GPFS 3.5 to 4.1.0.8 and it persists even after upgraded to GPFS 4.2.0.3 and the very latest sernet-samba. And to be clear, when we upgraded from GPFS 3.5 to 4.1 we did *not* upgrade SAMBA versions at that time. Therefore, I believe that something changed in GPFS. That doesn?t mean it?s GPFS? fault, of course. SAMBA may have been relying on a bugundocumented feature in GPFS that IBM fixed for all I know, and I?m obviously speculating here. The problem we see is that the .snapshots directory in each folder can be cd?d to but is empty. The snapshots are all there, however, if you: cd //.snapshots//rest/of/path/to/folder/in/question This obviously prevents users from being able to do their own recovery of files unless you do something like what you describe, which we are unwilling to do for security reasons. We have a ticket open with DDN? Kevin On Jun 20, 2016, at 8:45 AM, Sobey, Richard A > wrote: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don?t know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set ?force user = root? all the snapshots are available. I think the answer is SAMBA, right? We?re running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 20 16:03:50 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Jun 2016 11:03:50 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 20 16:03:50 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Jun 2016 11:03:50 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Mon Jun 20 16:10:13 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Mon, 20 Jun 2016 15:10:13 +0000 Subject: [gpfsug-discuss] ESS GL6 Message-ID: Couple of questions regarding Spectrum Scale 4.2 and ESS. We recently got our ESS delivered and are putting it in production this week. Previous to ESS we ran GPFS 3.5 and IBM DCS3700 storage arrays. My question about ESS and Spectrum Scale has to do with querying available free space and adding capacity to existing file system. In the old days of GPFS 3.5 I would create LUNs on 3700, zone them to appropriate hosts, and then see them as multipath devices on NSD servers. After that, I would create NSD disks and add them to the filesystem. With the ESS, however, I don't think the process is quite the same. IBM tech that was here installing the system has created all the "LUNs" or the equivalent in the ESS system. How do you I query what space is available to add to the existing filesystems, and then how do you actually add space? I am reading ESS RedBook but the answers are not obvious. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Jun 20 16:25:31 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 20 Jun 2016 17:25:31 +0200 Subject: [gpfsug-discuss] ESS GL6 In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Jun 20 17:29:15 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 20 Jun 2016 16:29:15 +0000 Subject: [gpfsug-discuss] ESS GL6 In-Reply-To: References: Message-ID: "mmlsrecoverygroup $name -L" will tell you how much raw capacity is left in a recoverygroup. You will then need to create a vdisk stanza file where you specify the capacity, blocksize, raidcode, etc. for the vdisk you want. (check "man mmcrvdisk") Then "mmcrvdisk stanzafile" to create the vdisk, and "mmcrnsd stanzafile" to create the NSDs. From then on it's standard gpfs. -jf man. 20. jun. 2016 kl. 17.25 skrev Olaf Weiser : > Hi Damir, > mmlsrecovergroup --> will show your RG > > mmlsrecoverygroup RG -L .. will provide capacity information > > or .. you can use the GUI > > with ESS / GNR , there's no need any more to create more than one > vdisk(=nsd) per RG for a pool > > a practical approach/example for you > so a file system consists of > 1 vdisk(NSD) for MetaData , RAID: 4WR , BS 1M in RG"left" > 1 vdisk(NSD) for MetaData , Raid : 4WR, BS 1M in RG "right" > 1 vdisk (NSD) for data , 8+3p , BS 1...16M .. depends on your > data/workload in RG "left" > 1 vdisk (NSD) for data , 8+3p , BS 1...16M .. depends on your > data/workload in RG "right" > > so 4 NSDs to provide everything you need to serve a file system .. > > > the size of the vdisks can be up to half of the capacity of your RG > > please note: if you come from an existing environment , and the file > system should be migrated to ESS (online) , you might hit some limitations > like > - blocksize (can not be changed) > - disk size.. depending on the existing storage pools/disk sizes.... > > > have fun > cheers > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert > Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, > HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Damir Krstic > To: gpfsug main discussion list > Date: 06/20/2016 05:10 PM > Subject: [gpfsug-discuss] ESS GL6 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Couple of questions regarding Spectrum Scale 4.2 and ESS. We recently got > our ESS delivered and are putting it in production this week. Previous to > ESS we ran GPFS 3.5 and IBM DCS3700 storage arrays. > > My question about ESS and Spectrum Scale has to do with querying available > free space and adding capacity to existing file system. > > In the old days of GPFS 3.5 I would create LUNs on 3700, zone them to > appropriate hosts, and then see them as multipath devices on NSD servers. > After that, I would create NSD disks and add them to the filesystem. > > With the ESS, however, I don't think the process is quite the same. IBM > tech that was here installing the system has created all the "LUNs" or the > equivalent in the ESS system. How do you I query what space is available to > add to the existing filesystems, and then how do you actually add space? > > I am reading ESS RedBook but the answers are not obvious. > > Thanks, > Damir _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Mon Jun 20 20:35:54 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Mon, 20 Jun 2016 19:35:54 +0000 Subject: [gpfsug-discuss] easiest way of restarting Spectrum Scale 4.2 web interface Message-ID: My ESS/Spectrum Scale web interface is spinning with "loading" screen without ever prompting me for username and password. I was able to log in early last week but something has happened since last week to make it stop working. Is there a quick way of reloading web interface to see if that fixes it before I submit help ticket with IBM? Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Jun 20 20:38:03 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 20 Jun 2016 19:38:03 +0000 Subject: [gpfsug-discuss] easiest way of restarting Spectrum Scale 4.2 web interface In-Reply-To: References: Message-ID: "systemctl restart gpfsgui" -jf man. 20. jun. 2016 kl. 21.36 skrev Damir Krstic : > My ESS/Spectrum Scale web interface is spinning with "loading" screen > without ever prompting me for username and password. I was able to log in > early last week but something has happened since last week to make it stop > working. > > Is there a quick way of reloading web interface to see if that fixes it > before I submit help ticket with IBM? > > Thanks, > Damir > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaurang.tapase at in.ibm.com Wed Jun 22 06:25:47 2016 From: gaurang.tapase at in.ibm.com (Gaurang Tapase) Date: Wed, 22 Jun 2016 10:55:47 +0530 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Message-ID: Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 22 11:33:40 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 22 Jun 2016 10:33:40 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: Message-ID: Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) -------------------------------------------------------------------------Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 22 11:39:06 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 22 Jun 2016 10:39:06 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: Message-ID: Does anyone in the field have much experience with using file heat for migration, whether for object or more generically? In particular using policies to move files both ways dependant on recent usage patterns. And also if you ever move files to colder storage without necessarily waiting until the files are say over 7 days old, since you know they are not going to be used for a while. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 12:34:11, daniel.kidger at uk.ibm.com wrote: From: daniel.kidger at uk.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: Date: 22 Jun 2016 12:34:11 Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AUUnless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Wed Jun 22 12:30:10 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 22 Jun 2016 11:30:10 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Message-ID: <1F32A0BF-7B3B-4407-B0D0-1F05A4B144ED@nuance.com> Hi Daniel I've done a fair amount of work with GPFS "File Heat" and policy migration here at Nuance. Let me know if you want to discuss it a bit more. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid robert.oesterlin @ nuance.com From: on behalf of Daniel Kidger Reply-To: gpfsug main discussion list Date: Wednesday, June 22, 2016 at 5:39 AM To: gpfsug main discussion list Cc: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Does anyone in the field have much experience with using file heat for migration, whether for object or more generically? In particular using policies to move files both ways dependant on recent usage patterns. And also if you ever move files to colder storage without necessarily waiting until the files are say over 7 days old, since you know they are not going to be used for a while. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse ________________________________ On 22 Jun 2016, 12:34:11, daniel.kidger at uk.ibm.com wrote: From: daniel.kidger at uk.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: Date: 22 Jun 2016 12:34:11 Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse ________________________________ On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Wed Jun 22 13:21:54 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Wed, 22 Jun 2016 13:21:54 +0100 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: References: Message-ID: <1466598114.3773.201.camel@buzzard.phy.strath.ac.uk> On Wed, 2016-06-22 at 10:39 +0000, Daniel Kidger wrote: > Does anyone in the field have much experience with using file heat for > migration, whether for object or more generically? In particular using > policies to move files both ways dependant on recent usage patterns. > In my experience moving files from slow to fast storage (I am ignoring a tape based or other really slow layer here) was generally a waste of time for three reasons. Firstly by the time you notice that the file is in use again 99% of the time user has stopped using it anyway so there is no advantage to moving it back to faster storage. You would need to scan the file system more than once a day to capture more reused files in my experience. Secondly if the data is modified and saved due to the way most software handles this you end up with a "new" file which will land in the fast storage anyway. That is most software on a save creates a new temporary file first with the new version of the file, renames the old file to something temporary, renames the new file to the right file name and then finally deletes the old version of file. That way if the program crashes somewhere in the save there will always be a good version of the file to go back to. Thirdly the slower tier (I am thinking along the lines of large RAID6 or equivalent devices) generally has lots of spare IOPS capacity anyway. Consequently the small amount or "revisited" data that does remain in use beyond a day is not going to have a significant impact. I measured around 10 IOPS on average on the slow disk in a general purpose file system. The only time it broke this was when a flush of data form the faster tier arrived at which point they would peg out at around 120 IOPS which is what you would expect for a 7200 RPM disk. > > And also if you ever move files to colder storage without necessarily > waiting until the files are say over 7 days old, since you know they > are not going to be used for a while. > I would do things like punt .iso images directly to slower storage. Those sorts of files generally don't benefit from having low seek times which is what your fast disk pools give you. I would suggest that video files like MPEG and MP4 could be potentially also be treated similarly. So a rule like the following /* force ISO images onto nearline storage */ RULE 'iso' SET POOL 'slow' WHERE LOWER(NAME) LIKE '%.iso' What you might punt to slower storage will likely depend heavily on what you file system is being used for. You can of course also use this to "discourage" your storage from being used for "personal" use by giving certain file types lower performance. So a rule like the following puts all the music files on slower storage. /* force MP3's and the like onto nearline storage forever */ RULE 'mp3' SET POOL 'slow' WHERE LOWER(NAME) LIKE '%.mp3' OR LOWER(NAME) LIKE '%.m4a' OR LOWER(NAME) LIKE '%.wma' JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From chekh at stanford.edu Fri Jun 24 19:09:59 2016 From: chekh at stanford.edu (Alex Chekholko) Date: Fri, 24 Jun 2016 11:09:59 -0700 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: References: Message-ID: <576D7777.50609@stanford.edu> On 06/21/2016 10:25 PM, Gaurang Tapase wrote: > Hello, > > Based on inputs from some of the Spectrum Scale users, sharing a blog > link for "Automatic tiering of object data based on object heat" by Smita. > https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ > Thanks for the blog post. I wanted to post my experiences: Note "sata0" is really NL-SAS. [root at scg-gs0 ~]# df -h|grep gsfs /dev/gsfs0 4.4P 3.1P 1.4P 70% /srv/gsfs0 [root at scg-gs0 ~]# df -i|grep gsfs /dev/gsfs0 314572800 234237984 80334816 75% /srv/gsfs0 [root at scg-gs0 ~]# mmlspool gsfs0 Storage pools in file system at '/srv/gsfs0': Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB) system 0 256 KB yes yes 1887436800 118287616 ( 6%) 1887436800 328248832 ( 17%) sas0 65537 256 KB yes no 259795189760 59495012864 ( 23%) 0 0 ( 0%) sata0 65538 256 KB yes no 4523875631104 1405369561600 ( 31%) 0 0 ( 0%) [root at scg-gs0 ~]# mmlspolicy gsfs0 -L RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' [root at scg-gs0 ~]# cat policy_gpool.txt rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' LIMIT(80) THEN 'sata0' rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT) [root at scg-gs0 ~]# mmlsconfig|grep -i heat fileHeatPeriodMinutes 1440 fileHeatLossPercent 1 The first thing to know is that the MIGRATE rules only run when you run 'mmapplypolicy', even is you have the migrate policy loaded. So you have to decide when/how to do that. The filesystem scan takes a while; here are the parameters that let a full scan of our filesystem complete in under 10mins (8 I/O servers, ~300M inodes): mmapplypolicy gsfs0 -P policy.txt -N scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g /srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4 The issues we have are: 1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD, 259TB 10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% ?fast?, 94.57% ?slow?) 2) the client I/O can max out the IOPS in the "slow" pool and in the "fast" pool The original idea was to have all client I/O go to the 10K pool, then migrate least-used data to slow pool. Unfortunately our fast pool fills up too quickly and then GPFS can't empty it quickly enough into the slow pool. So I end up switching the placement policy to put files into the slow pool. And then emptying the fast pool into the slow pool doesn't work too well because the slow pool is already maxed out on IOPS just handling the workload. So moving the data from fast to slow takes weeks and impacts performance. The original intention was: [root at scg-gs0 ~]# cat policy_placement.txt RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%' RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' I would recommend that your total workload fits in both capacity and performance into your "primary" aka "fast" pool. I also just learned recently of the LIMIT parameter for placement, e.g. RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' LIMIT (95) RULE 'spillover' SET POOL 'sata0' So that would help with the pool not filling, but won't help with not having enough I/O for migration+clients. Regards, -- Alex Chekholko chekh at stanford.edu From kraemerf at de.ibm.com Sun Jun 26 12:04:42 2016 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Sun, 26 Jun 2016 13:04:42 +0200 Subject: [gpfsug-discuss] FIY - Efficient data tiering with the Swift High Latency Middleware (SwiftHLM) Message-ID: https://developer.ibm.com/open/2016/06/21/efficient-data-tiering-swift-high-latency-middleware-swifthlm/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Sun Jun 26 16:22:46 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Sun, 26 Jun 2016 15:22:46 +0000 Subject: [gpfsug-discuss] verbs rdma errors in logs Message-ID: We recently enabled verbs/rdma on our IB network (previously we used IPoIB exclusively) and now are getting all sorts of errors/warning in logs: Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA rdma read error IBV_WC_RETRY_EXC_ERR to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 vendor_err 129 Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 1589 Jun 25 20:40:05 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.124.12 (qnode4054-ib0.quest) on mlx5_0 port 1 fabnum 0 index 1417 Jun 25 18:30:01 ems1 root: mmfs: [N] VERBS RDMA closed connection to 172.41.130.131 (qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 195 Jun 25 18:28:23 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.130.131 (qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 1044 Something to note (not sure if this is important or not) is that our ESS storage cluster and our login nodes are in connected mode with 64K MTU and all compute nodes are in datagram mode with 2.4K MTU. Are these messages something to be concerned about? Cluster seems to be performing well and although there are some node ejections, they do not seem higher than before we turned on verbs/rdma. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sun Jun 26 17:14:10 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Sun, 26 Jun 2016 16:14:10 +0000 Subject: [gpfsug-discuss] verbs rdma errors in logs References: [gpfsug-discuss] verbs rdma errors in logs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101C368BF@NDMSMBX404.ndc.nasa.gov> Damir, We see similar things in our environment (3.5k nodes) that seem to correlate with GPFS recovery events. I did some digging at it seemed to me that these errors more or less mean the other side of the VERBS connection hung up on the other. The message format seems a little alarming but I think it's innocuous. I'm curious to hear what others have to say. -Aaron From: Damir Krstic Sent: 6/26/16, 11:23 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] verbs rdma errors in logs We recently enabled verbs/rdma on our IB network (previously we used IPoIB exclusively) and now are getting all sorts of errors/warning in logs: Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA rdma read error IBV_WC_RETRY_EXC_ERR to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 vendor_err 129 Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 1589 Jun 25 20:40:05 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.124.12 (qnode4054-ib0.quest) on mlx5_0 port 1 fabnum 0 index 1417 Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 195 Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 1044 Something to note (not sure if this is important or not) is that our ESS storage cluster and our login nodes are in connected mode with 64K MTU and all compute nodes are in datagram mode with 2.4K MTU. Are these messages something to be concerned about? Cluster seems to be performing well and although there are some node ejections, they do not seem higher than before we turned on verbs/rdma. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 27 15:49:27 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 27 Jun 2016 10:49:27 -0400 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: <576D7777.50609@stanford.edu> References: <576D7777.50609@stanford.edu> Message-ID: I'd like to see comments and/or questions and/or suggestions from any others who have implemented, deployed or seriously planned using FILE_HEAT and/or GROUP POOL. We implemented this a few years ago. Since we haven't had much feedback, it's either perfect or ... ? ;-) Alex, my reading of your story is that your hardware does not match your requirements, capacity, speeds, feeds. One new possibly helpful improvement is that SS release 4.2 includes QOSio which allows one to control the IOPs consumed by migration. So you can control the performance "hit" of running mmapplypolicy. You can even control it on-the-fly without killing a long running job -- for example you can throttle it back to 100iops during prime-time day shift operations -- marc of GPFS. From: Alex Chekholko To: gpfsug-discuss at spectrumscale.org Date: 06/24/2016 02:10 PM Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Sent by: gpfsug-discuss-bounces at spectrumscale.org On 06/21/2016 10:25 PM, Gaurang Tapase wrote: > Hello, > > Based on inputs from some of the Spectrum Scale users, sharing a blog > link for "Automatic tiering of object data based on object heat" by Smita. > https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ > Thanks for the blog post. I wanted to post my experiences: Note "sata0" is really NL-SAS. [root at scg-gs0 ~]# df -h|grep gsfs /dev/gsfs0 4.4P 3.1P 1.4P 70% /srv/gsfs0 [root at scg-gs0 ~]# df -i|grep gsfs /dev/gsfs0 314572800 234237984 80334816 75% /srv/gsfs0 [root at scg-gs0 ~]# mmlspool gsfs0 Storage pools in file system at '/srv/gsfs0': Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB) system 0 256 KB yes yes 1887436800 118287616 ( 6%) 1887436800 328248832 ( 17%) sas0 65537 256 KB yes no 259795189760 59495012864 ( 23%) 0 0 ( 0%) sata0 65538 256 KB yes no 4523875631104 1405369561600 ( 31%) 0 0 ( 0%) [root at scg-gs0 ~]# mmlspolicy gsfs0 -L RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' [root at scg-gs0 ~]# cat policy_gpool.txt rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' LIMIT(80) THEN 'sata0' rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT) [root at scg-gs0 ~]# mmlsconfig|grep -i heat fileHeatPeriodMinutes 1440 fileHeatLossPercent 1 The first thing to know is that the MIGRATE rules only run when you run 'mmapplypolicy', even is you have the migrate policy loaded. So you have to decide when/how to do that. The filesystem scan takes a while; here are the parameters that let a full scan of our filesystem complete in under 10mins (8 I/O servers, ~300M inodes): mmapplypolicy gsfs0 -P policy.txt -N scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g /srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4 The issues we have are: 1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD, 259TB 10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% ?fast?, 94.57% ?slow?) 2) the client I/O can max out the IOPS in the "slow" pool and in the "fast" pool The original idea was to have all client I/O go to the 10K pool, then migrate least-used data to slow pool. Unfortunately our fast pool fills up too quickly and then GPFS can't empty it quickly enough into the slow pool. So I end up switching the placement policy to put files into the slow pool. And then emptying the fast pool into the slow pool doesn't work too well because the slow pool is already maxed out on IOPS just handling the workload. So moving the data from fast to slow takes weeks and impacts performance. The original intention was: [root at scg-gs0 ~]# cat policy_placement.txt RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%' RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' I would recommend that your total workload fits in both capacity and performance into your "primary" aka "fast" pool. I also just learned recently of the LIMIT parameter for placement, e.g. RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' LIMIT (95) RULE 'spillover' SET POOL 'sata0' So that would help with the pool not filling, but won't help with not having enough I/O for migration+clients. Regards, -- Alex Chekholko chekh at stanford.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Mon Jun 27 17:22:25 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Mon, 27 Jun 2016 17:22:25 +0100 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: <577152C1.8070802@pixitmedia.com> Hi Marc, Quite. We have had coverage of (1) and (2) in the API for some time. My overriding thought is that (QoS aside) if one executes an mmapplypolicy, for whatever reason, then a certain amount of resource is consumed. This may not be preferred. So let me rephrase to the users: in your real-world working environment, would you prefer: a) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc which is a sanity check, but not a policy run, hence no load b) mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. To be clear; you can currently achieve either launch with the API as it stands. However we'd like to know what the general concensus would prefer to be the norm in such an implementation. Regarding dry run functionality this can be achieved globally as follows: setDryRun(True) or as a more granular decorator: @dryrun ... def delete_filesets(): Jez On 20/06/16 16:03, Marc A Kaplan wrote: > *Jez,* > > Regarding your recent post. Do the mmchpolicy and mmapplypolicy > commands have sufficient functionality for your purposes? > Are you suggesting some improvements? If so, what? Please provide > examples and/or specific suggestions. > > WRT your numbered items: > > (1) `mmchpolicy fsname -I policy-rules-file test` does a complete > syntax check on policy-rules-file and some other sanity checking. > > (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I > test -L 0` is another way to validate policy-rules-file. Rules like > MIGRATE and LIST that are interpreted by mmapplypolicy will be > scrutinized. For example check that each named pool is defined either > within the file system or by an EXTERNAL POOL rules. > > (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 > [other options]` will do a dry run and show what actions would be > performed on each file. > > > --marc > > ----------- > > API calls for Policies have the ability to run in 'test' or 'run' > mode. (ref: man mmapplypolicy) > > How would you expect to use dry-run functionality on a policy? > We have our own opinion, but we'd like to hear yours. > > 1) Validate the policy and print the policy content to stdout/other. > No mmapplypolicy is performed. > 2) Validate the policy and enforce mmapplypolicy with -I test > 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol > --flag1 --flag2 --etc > 4) Other > > Best regards, > > Jez > -- > Jez Tucker > Head of Research & Development > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media www.pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 27 18:35:33 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 27 Jun 2016 13:35:33 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: <577152C1.8070802@pixitmedia.com> References: <57598963.3040706@pixitmedia.com> <577152C1.8070802@pixitmedia.com> Message-ID: What I always recommend is to make a directory of test cases and run mmapplypolicy /gpfs-path/test-cases -I test (or -I yes if you want to try operating on the test files) -L 2 (or -L 6 or something in between) -P policy-rules [other-options] Because as I think you're saying mmapplypolicy gpfs-fsname -I test ... Can consume a lot of compute and IO resource, just scanning the metadata. From: Jez Tucker To: gpfsug main discussion list Date: 06/27/2016 12:22 PM Subject: Re: [gpfsug-discuss] Dry-Run functionality ? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Marc, Quite. We have had coverage of (1) and (2) in the API for some time. My overriding thought is that (QoS aside) if one executes an mmapplypolicy, for whatever reason, then a certain amount of resource is consumed. This may not be preferred. So let me rephrase to the users: in your real-world working environment, would you prefer: a) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc which is a sanity check, but not a policy run, hence no load b) mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. To be clear; you can currently achieve either launch with the API as it stands. However we'd like to know what the general concensus would prefer to be the norm in such an implementation. Regarding dry run functionality this can be achieved globally as follows: setDryRun(True) or as a more granular decorator: @dryrun ... def delete_filesets(): Jez On 20/06/16 16:03, Marc A Kaplan wrote: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media www.pixitmedia.com This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Wed Jun 29 16:38:09 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Wed, 29 Jun 2016 17:38:09 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Message-ID: <5773EB61.2040203@ugent.be> Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth From Robert.Oesterlin at nuance.com Wed Jun 29 16:41:11 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 29 Jun 2016 15:41:11 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Message-ID: Nothing "official" from IBM, but I believe it's been delayed until later in July. Not sure if IBM would comment publically on it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Kenneth Waegeman Reply-To: gpfsug main discussion list Date: Wednesday, June 29, 2016 at 10:38 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fudqCYcU99iTF3_mD3_zr_XvAB9BGNRjbIka32m0Kdw&s=GIVzGr_ag_wl2INE5BTuO1fDQ0QOfKdo--ljaxj6FcI&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From raot at bnl.gov Wed Jun 29 16:43:25 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 11:43:25 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy Message-ID: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 29 16:46:57 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 29 Jun 2016 15:46:57 +0000 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: References: Message-ID: Hi Tejas, One thing that I have had occasional success with is a lazy unmount - i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does help in my experience, but it?s one thing I try before bouncing a node. HTHAL? Kevin On Jun 29, 2016, at 10:43 AM, Tejas Rao > wrote: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raot at bnl.gov Wed Jun 29 16:58:09 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 11:58:09 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: References: Message-ID: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> "umount -l" returns with "not mounted" message. See below. [root@ ~]# umount -l /gpfs01 umount: /gpfs01: not mounted [root@ ~]# umount -l /dev/gpfs01 umount: /dev/gpfs01: not mounted [root@ ~]# [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Thanks. On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: > Hi Tejas, > > One thing that I have had occasional success with is a lazy unmount - > i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does > help in my experience, but it?s one thing I try before bouncing a > node. HTHAL? > > Kevin > >> On Jun 29, 2016, at 10:43 AM, Tejas Rao > > wrote: >> >> Hi all, >> >> I cannot get the filesystem mounted again on few nodes. >> >> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >> mount: /dev/gpfs01 already mounted or /gpfs01 busy >> mmmount: Command failed. Examine previous error messages to determine >> cause. >> >> >> >> Things I have tried and checked. >> >> >> >> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >> /dev/gpfs01 and /gpfs01 are not seen anywhere. >> >> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >> See below. >> >> 3. mmumount gpfs01 -f >> >> >> A reboot of the client would probably fix it but as the client is >> used in production, this is not easy to do. >> >> >> Are there any tricks/hacks to remount the filesystem again without >> rebooting the client. >> >> >> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >> GPFS file systems >> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >> Shutting down! >> 'shutdown' command about to kill process 11431 >> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >> mmfsenv: Module mmfslinux is still in use. >> Unmount all GPFS file systems and issue the command: >> mmfsadm cleanup >> Unloading module mmfs26 >> ERROR: Module mmfs26 is in use >> mmfsenv: Error unloading module mmfs26. >> Unloading module mmfslinux >> ERROR: Module mmfslinux is in use by mmfs26 >> mmfsenv: Error unloading module mmfslinux. >> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >> [root@~]# >> >> Thanks. >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Jun 29 17:02:27 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Jun 2016 16:02:27 +0000 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> References: , <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> Message-ID: Process running which is either from gpfs or using it. We used to see it when we ran torque from gpfs and forgot to stop it before shutting down gpfs. Or if you have dsmrecalld (for example) running on the node. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Tejas Rao [raot at bnl.gov] Sent: 29 June 2016 16:58 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy "umount -l" returns with "not mounted" message. See below. [root@ ~]# umount -l /gpfs01 umount: /gpfs01: not mounted [root@ ~]# umount -l /dev/gpfs01 umount: /dev/gpfs01: not mounted [root@ ~]# [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Thanks. On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: Hi Tejas, One thing that I have had occasional success with is a lazy unmount - i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does help in my experience, but it?s one thing I try before bouncing a node. HTHAL? Kevin On Jun 29, 2016, at 10:43 AM, Tejas Rao <raot at bnl.gov> wrote: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From konstantin.arnold at unibas.ch Wed Jun 29 17:05:33 2016 From: konstantin.arnold at unibas.ch (Konstantin Arnold) Date: Wed, 29 Jun 2016 18:05:33 +0200 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> References: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> Message-ID: <5773F1CD.2000205@unibas.ch> Hi Tejas, we have observed this behavior too, typically when automounter or open file handles are involved. We tried to close the file handles (kill the process that has them open), next tried to unmount the automouter path and then mmumount filesystem again. In cases where we could not find and close the open file handles (e.g. process cannot be killed anymore) or you're stuck with unmounting the autoumounter path, we had to stop GPFS and verify the kernel module gets unloaded. In case the kernel module stays loaded, our only option was to reboot the node. Best Konstantin On 06/29/2016 05:58 PM, Tejas Rao wrote: > "umount -l" returns with "not mounted" message. See below. > > > [root@ ~]# umount -l /gpfs01 > umount: /gpfs01: not mounted > > [root@ ~]# umount -l /dev/gpfs01 > umount: /dev/gpfs01: not mounted > [root@ ~]# > > > [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 > Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... > mount: /dev/gpfs01 already mounted or /gpfs01 busy > mmmount: Command failed. Examine previous error messages to determine cause. > > Thanks. > > > On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: >> Hi Tejas, >> >> One thing that I have had occasional success with is a lazy unmount - >> i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does >> help in my experience, but it?s one thing I try before bouncing a >> node. HTHAL? >> >> Kevin >> >>> On Jun 29, 2016, at 10:43 AM, Tejas Rao >>> <raot at bnl.gov> wrote: >>> >>> Hi all, >>> >>> I cannot get the filesystem mounted again on few nodes. >>> >>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >>> mount: /dev/gpfs01 already mounted or /gpfs01 busy >>> mmmount: Command failed. Examine previous error messages to determine >>> cause. >>> >>> >>> >>> Things I have tried and checked. >>> >>> >>> >>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >>> /dev/gpfs01 and /gpfs01 are not seen anywhere. >>> >>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >>> See below. >>> >>> 3. mmumount gpfs01 -f >>> >>> >>> A reboot of the client would probably fix it but as the client is >>> used in production, this is not easy to do. >>> >>> >>> Are there any tricks/hacks to remount the filesystem again without >>> rebooting the client. >>> >>> >>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >>> GPFS file systems >>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >>> Shutting down! >>> 'shutdown' command about to kill process 11431 >>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >>> mmfsenv: Module mmfslinux is still in use. >>> Unmount all GPFS file systems and issue the command: >>> mmfsadm cleanup >>> Unloading module mmfs26 >>> ERROR: Module mmfs26 is in use >>> mmfsenv: Error unloading module mmfs26. >>> Unloading module mmfslinux >>> ERROR: Module mmfslinux is in use by mmfs26 >>> mmfsenv: Error unloading module mmfslinux. >>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >>> [root@~]# >>> >>> Thanks. >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- ------------------------------------------------------------------------------------------- Konstantin Arnold | University of Basel & SIB Klingelbergstrasse 50/70 | CH-4056 Basel | Phone: +41 61 267 15 82 Email: konstantin.arnold at unibas.ch From raot at bnl.gov Wed Jun 29 17:13:10 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 12:13:10 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <5773F1CD.2000205@unibas.ch> References: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> <5773F1CD.2000205@unibas.ch> Message-ID: Yes, open file handles were probably the issue, although I did not see them in lsof. I restarted my application and now I can mount /gpfs01/ again. Thanks, Tejas. On 6/29/2016 12:05, Konstantin Arnold wrote: > Hi Tejas, > > we have observed this behavior too, typically when automounter or open > file handles are involved. We tried to close the file handles (kill > the process that has them open), next tried to unmount the automouter > path and then mmumount filesystem again. In cases where we could not > find and close the open file handles (e.g. process cannot be killed > anymore) or you're stuck with unmounting the autoumounter path, we had > to stop GPFS and verify the kernel module gets unloaded. In case the > kernel module stays loaded, our only option was to reboot the node. > > Best > Konstantin > > > > On 06/29/2016 05:58 PM, Tejas Rao wrote: >> "umount -l" returns with "not mounted" message. See below. >> >> >> [root@ ~]# umount -l /gpfs01 >> umount: /gpfs01: not mounted >> >> [root@ ~]# umount -l /dev/gpfs01 >> umount: /dev/gpfs01: not mounted >> [root@ ~]# >> >> >> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >> Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... >> mount: /dev/gpfs01 already mounted or /gpfs01 busy >> mmmount: Command failed. Examine previous error messages to determine >> cause. >> >> Thanks. >> >> >> On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: >>> Hi Tejas, >>> >>> One thing that I have had occasional success with is a lazy unmount - >>> i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does >>> help in my experience, but it?s one thing I try before bouncing a >>> node. HTHAL? >>> >>> Kevin >>> >>>> On Jun 29, 2016, at 10:43 AM, Tejas Rao >>>> <raot at bnl.gov> wrote: >>>> >>>> Hi all, >>>> >>>> I cannot get the filesystem mounted again on few nodes. >>>> >>>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >>>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >>>> mount: /dev/gpfs01 already mounted or /gpfs01 busy >>>> mmmount: Command failed. Examine previous error messages to determine >>>> cause. >>>> >>>> >>>> >>>> Things I have tried and checked. >>>> >>>> >>>> >>>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >>>> /dev/gpfs01 and /gpfs01 are not seen anywhere. >>>> >>>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >>>> See below. >>>> >>>> 3. mmumount gpfs01 -f >>>> >>>> >>>> A reboot of the client would probably fix it but as the client is >>>> used in production, this is not easy to do. >>>> >>>> >>>> Are there any tricks/hacks to remount the filesystem again without >>>> rebooting the client. >>>> >>>> >>>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >>>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >>>> GPFS file systems >>>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >>>> Shutting down! >>>> 'shutdown' command about to kill process 11431 >>>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >>>> mmfsenv: Module mmfslinux is still in use. >>>> Unmount all GPFS file systems and issue the command: >>>> mmfsadm cleanup >>>> Unloading module mmfs26 >>>> ERROR: Module mmfs26 is in use >>>> mmfsenv: Error unloading module mmfs26. >>>> Unloading module mmfslinux >>>> ERROR: Module mmfslinux is in use by mmfs26 >>>> mmfsenv: Error unloading module mmfslinux. >>>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >>>> [root@~]# >>>> >>>> Thanks. >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> ? >>> Kevin Buterbaugh - Senior System Administrator >>> Vanderbilt University - Advanced Computing Center for Research and >>> Education >>> Kevin.Buterbaugh at vanderbilt.edu >>> - (615)875-9633 >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > From S.J.Thompson at bham.ac.uk Wed Jun 29 17:18:50 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Jun 2016 16:18:50 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: <5773EB61.2040203@ugent.be> References: <5773EB61.2040203@ugent.be> Message-ID: I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From daniel.kidger at uk.ibm.com Wed Jun 29 17:25:07 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 29 Jun 2016 16:25:07 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: Message-ID: Informally I hear that we a still a couple of weeks away. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 29 Jun 2016, 18:19:07, S.J.Thompson at bham.ac.uk wrote: From: S.J.Thompson at bham.ac.uk To: gpfsug-discuss at spectrumscale.org Cc: Date: 29 Jun 2016 18:19:07 Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discussUnless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Thu Jun 30 11:46:16 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Thu, 30 Jun 2016 12:46:16 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: References: <5773EB61.2040203@ugent.be> Message-ID: The GA date moved to July 29th. A gating issue was found late during the testing and nobody wanted to risk to release the product with this issue and provide enough time to make sure the issue is fixed and well tested. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect /PMP - Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-6131-84-3465 IBM Deutschland Mobile: +49-170-6346601 Hechtsheimer Str. 2 E-Mail: stschmid at de.ibm.com 55131 Mainz Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 29.06.2016 18:19 Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Thu Jun 30 11:56:04 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Thu, 30 Jun 2016 12:56:04 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: References: <5773EB61.2040203@ugent.be> Message-ID: <5774FAC4.5060705@ugent.be> Thanks for all the info, it's good to know! Kind regards, Kenneth On 30/06/16 12:46, Stefan Schmidt wrote: > The GA date moved to July 29th. A gating issue was found late during > the testing and nobody wanted to risk to release the product with this > issue and provide enough time to make sure the issue is fixed and well > tested. > > Mit freundlichen Gr??en / Kind regards > > *Stefan Schmidt* > > Scrum Master IBM Spectrum Scale GUI / Senior IT Architect /PMP - Dept. > M069 / IBM Spectrum Scale Software Development > > IBM Systems Group > > IBM Deutschland > > ------------------------------------------------------------------------ > > Phone: +49-6131-84-3465 IBM Deutschland > Mobile: +49-170-6346601 Hechtsheimer Str. 2 > E-Mail: stschmid at de.ibm.com 55131 Mainz > > > Germany > ------------------------------------------------------------------------ > > IBM Deutschland Research & Development GmbH / Vorsitzende des > Aufsichtsrats: Martina Koederitz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht > Stuttgart, HRB 243294 > > > > > > > > From: "Simon Thompson (Research Computing - IT Services)" > > To: gpfsug main discussion list > Date: 29.06.2016 18:19 > Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > > I'm not sure they announced... Someone may have said that they > expected June, you did read all the disclaimer in the slide decks right ;) > > But seriously, its not a formal product announcement platform, so > whilst people may talk about features, ideas etc... It shouldn't be > held as a commitment. > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth > Waegeman [kenneth.waegeman at ugent.be] > Sent: 29 June 2016 16:38 > To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 > > Hi all, > > On the User meeting in London a month ago, they announced that SS 4.2.1 > would be released somewhere in June. Since I did not found any release > notes or packages at the passport advantage portal, I guess it it is not > yet released? Does someone knows what would be the timing for this? > > Thank you! > > Cheers, > Kenneth > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From service at metamodul.com Thu Jun 30 12:53:11 2016 From: service at metamodul.com (- -) Date: Thu, 30 Jun 2016 13:53:11 +0200 (CEST) Subject: [gpfsug-discuss] Generic HA Environment Message-ID: <743599299.124585.fca32d27-f514-4edd-af56-0af407bd9929.open-xchange@email.1und1.de> Hi IBM folks, i would like to know if the current High Available (HA) infrastructure used by the Cluster Export Services (ces) like SMB & NFS can be used for private extensions meaning i am able to add additional service by my own. Thus there would be no need to write my own HA Infrastructre. tia Hajo From Mark.Bush at siriuscom.com Thu Jun 30 15:01:46 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Jun 2016 14:01:46 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store Message-ID: I know in the trial VM it recommends using CyberDuck to get to the object side of scale. Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush | Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions | www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Jun 30 15:29:17 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 30 Jun 2016 14:29:17 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: The object side is just openstack swift, so f.ex: https://github.com/openstack/python-swiftclient/blob/master/README.rst -jf tor. 30. jun. 2016 kl. 16.01 skrev Mark.Bush at siriuscom.com < Mark.Bush at siriuscom.com>: > I know in the trial VM it recommends using CyberDuck to get to the object > side of scale. Does anyone have a programmatic example of doing this? > Python would be preferred here. > > > > Mark R. Bush | Solutions Architect > Mobile: 210.237.8415 | mark.bush at siriuscom.com > Sirius Computer Solutions | www.siriuscom.com > 10100 Reunion Place, Suite 500, San Antonio, TX 78216 > > > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, and > exempt from disclosure under applicable law. If you are not the intended > recipient, you are hereby notified that any use, dissemination, > distribution, or copying of this communication is strictly prohibited. This > message may be viewed by parties at Sirius Computer Solutions other than > those named in the message header. This message does not contain an > official representation of Sirius Computer Solutions. If you have received > this communication in error, notify Sirius Computer Solutions immediately > and (i) destroy this message if a facsimile or (ii) delete this message > immediately if this is an electronic communication. Thank you. > Sirius Computer Solutions > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Thu Jun 30 15:33:31 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Thu, 30 Jun 2016 15:33:31 +0100 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: <57752DBB.3070403@pixitmedia.com> Hi Depending what your endpoint object store type is... https://github.com/openstack/python-swiftclient http://boto.cloudhackers.com/en/latest/ Though, I highly recommend: https://libcloud.apache.org/ I.E. http://libcloud.readthedocs.io/en/latest/storage/examples.html Best, Jez On 30/06/16 15:01, Mark.Bush at siriuscom.com wrote: > > I know in the trial VM it recommends using CyberDuck to get to the > object side of scale. Does anyone have a programmatic example of > doing this? Python would be preferred here. > > Mark R. Bush | Solutions Architect > Mobile: 210.237.8415 | mark.bush at siriuscom.com > Sirius Computer Solutions | www.siriuscom.com > 10100 Reunion Place, Suite 500, San Antonio, TX 78216 > > This message (including any attachments) is intended only for the use > of the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, > dissemination, distribution, or copying of this communication is > strictly prohibited. This message may be viewed by parties at Sirius > Computer Solutions other than those named in the message header. This > message does not contain an official representation of Sirius Computer > Solutions. If you have received this communication in error, notify > Sirius Computer Solutions immediately and (i) destroy this message if > a facsimile or (ii) delete this message immediately if this is an > electronic communication. Thank you. > > Sirius Computer Solutions > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media / ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Jun 30 15:48:00 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Jun 2016 14:48:00 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: <57752DBB.3070403@pixitmedia.com> References: <57752DBB.3070403@pixitmedia.com> Message-ID: <19124267-08D0-4232-BBF0-835653D8D4BB@siriuscom.com> Thanks to Jan-Frode and Jez. I will use the import swiftclient mechanism for this. From: Jez Tucker Organization: Pixit Media Limited Reply-To: gpfsug main discussion list Date: Thursday, June 30, 2016 at 9:33 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] restful way to get at SS OBJ store Hi Depending what your endpoint object store type is... https://github.com/openstack/python-swiftclient http://boto.cloudhackers.com/en/latest/ Though, I highly recommend: https://libcloud.apache.org/ I.E. http://libcloud.readthedocs.io/en/latest/storage/examples.html Best, Jez On 30/06/16 15:01, Mark.Bush at siriuscom.com wrote: I know in the trial VM it recommends using CyberDuck to get to the object side of scale. Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush | Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions | www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media / ArcaStream www.pixitmedia.com www.arcastream.com [http://pixitmedia.com/sig/sig-cio.jpg] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Thu Jun 30 16:37:05 2016 From: billowen at us.ibm.com (Bill Owen) Date: Thu, 30 Jun 2016 08:37:05 -0700 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: Also see: http://docs.openstack.org/developer/swift/api/object_api_v1_overview.html The trial VM has python-swiftclient installed. You can execute basic swift commands as follows: source openrc #define environment variables for account, user credentials etc. swift stat #show account statistics swift upload One helpful trick is to execute the client commands with --debug flag. This will show the exact http request that the client code is sending. For example: # swift --debug stat INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): client28 DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_7c0243a523cf436e9a0e79c9b7083f9e HTTP/1.1" 204 0 DEBUG:swiftclient:REQ: curl -i http://client28:8080/v1/AUTH_7c0243a523cf436e9a0e79c9b7083f9e -I -H "X-Auth-Token: " DEBUG:swiftclient:RESP STATUS: 204 No Content DEBUG:swiftclient:RESP HEADERS: [('content-length', '0'), ('x-account-object-count', '0'), ('x-account-project-domain-id', 'default'), ('x-account-storage-policy-policy-0-bytes-used', '0'), ('x-account-storage-policy-policy-0-container-count', '10'), ('x-timestamp', '1467238461.63001'), ('x-account-storage-policy-policy-0-object-count', '0'), ('x-trans-id', 'txd3e3d9ca87fe4e23a5c9c-0057753b57'), ('date', 'Thu, 30 Jun 2016 15:31:36 GMT'), ('x-account-bytes-used', '0'), ('x-account-container-count', '10'), ('content-type', 'text/plain; charset=utf-8'), ('accept-ranges', 'bytes')] Account: AUTH_7c0243a523cf436e9a0e79c9b7083f9e Containers: 10 Objects: 0 Bytes: 0 Containers in policy "policy-0": 10 Objects in policy "policy-0": 0 Bytes in policy "policy-0": 0 X-Account-Project-Domain-Id: default X-Timestamp: 1467238461.63001 X-Trans-Id: txd3e3d9ca87fe4e23a5c9c-0057753b57 Content-Type: text/plain; charset=utf-8 Accept-Ranges: bytes Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: Jan-Frode Myklebust To: "gpfsug-discuss at spectrumscale.org" Date: 06/30/2016 07:29 AM Subject: Re: [gpfsug-discuss] restful way to get at SS OBJ store Sent by: gpfsug-discuss-bounces at spectrumscale.org The object side is just openstack swift, so f.ex: https://github.com/openstack/python-swiftclient/blob/master/README.rst -jf tor. 30. jun. 2016 kl. 16.01 skrev Mark.Bush at siriuscom.com < Mark.Bush at siriuscom.com>: I know in the trial VM it recommends using CyberDuck to get to the object side of scale.? Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush?| Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions?|?www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Wed Jun 1 15:55:57 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 1 Jun 2016 14:55:57 +0000 Subject: [gpfsug-discuss] Building portability layer fails In-Reply-To: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk> References: <16F6CB71-3BC1-4F9F-BA2F-DC6911325C8A@qsplace.co.uk>, <20160528004711.1B4A46A03D@b03ledav003.gho.boulder.ibm.com> Message-ID: <201606011456.u51EuIRY007236@d06av01.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 1 16:03:04 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 1 Jun 2016 15:03:04 +0000 Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All versions In-Reply-To: References: Message-ID: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Wed Jun 1 16:21:28 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Wed, 01 Jun 2016 16:21:28 +0100 Subject: [gpfsug-discuss] GPFS/Spectrum Scale security vulernability - All versions In-Reply-To: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> References: <201606011505.u51F5QrJ003407@d06av03.portsmouth.uk.ibm.com> Message-ID: <1464794488.3773.15.camel@buzzard.phy.strath.ac.uk> On Wed, 2016-06-01 at 15:03 +0000, Daniel Kidger wrote: > By the way the files affected are all binaries so not as obvious a > security risk as allowing a backdoor to an interactive ksh shell as > root. Well yeah, Linux has not allowed setuid on scripts of any kind for around 20 years now. Hence even if you are running on AIX they will be binaries because otherwise you would have to maintain a script for AIX and a binary for Linux. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From douglasof at us.ibm.com Wed Jun 1 20:54:04 2016 From: douglasof at us.ibm.com (Douglas O'flaherty) Date: Wed, 1 Jun 2016 15:54:04 -0400 Subject: [gpfsug-discuss] IBM Webinar Message-ID: <20160601195415.A2EA26E044@b03ledav001.gho.boulder.ibm.com> Not everyone can make it to a User Group, so we are kicking off a series of webinars with IBM Spectrum Scale experts. Join us for the first one - and bring questions. I will be hosting and having two of our architects on the call. Title: Building a Better Data Ocean: what's new in IBM Spectrum Scale Date: June 16, 12:00 EST Speakers: Piyush Chaudhary, Spectrum Scale Architect, Big Data & Analytics Rob Basham Transparent Cloud tiering and Cloud archive Storage lead architect URL: https://attendee.gotowebinar.com/register/6431264887578253057 -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Wed Jun 1 23:04:45 2016 From: billowen at us.ibm.com (Bill Owen) Date: Wed, 1 Jun 2016 15:04:45 -0700 Subject: [gpfsug-discuss] SECOND CHANCE - OpenStack & Spectrum Scale Usage Survey In-Reply-To: References: Message-ID: <20160601220453.5CD0EC604C@b03ledav006.gho.boulder.ibm.com> If you already filled out the OpenStack & Spectrum Scale Usage Survey, Thank You! If you haven't yet, there's still time to be heard - Please take 10 minutes to answer the questions in this short survey: https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey Thank you, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: Bill Owen/Tucson/IBM To: gpfsug-discuss at gpfsug.org Date: 05/14/2016 05:05 AM Subject: OpenStack & Spectrum Scale Usage Survey We in Spectrum Scale development are very interested in learning about your current & planned usage of OpenStack with Spectrum Scale, as well as plans for deploying containers in your Spectrum Scale environment. We would like to use this information to help shape our road map in this area over the next 18-24 months. Please take 10 minutes to answer the questions in this short survey: https://www.surveygizmo.com/s3/2774614/IBMSpectrumScale-OpenStackUsageSurvey Thank you, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Paul.Sanchez at deshaw.com Fri Jun 3 14:38:01 2016 From: Paul.Sanchez at deshaw.com (Sanchez, Paul) Date: Fri, 3 Jun 2016 13:38:01 +0000 Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 on Haswell/Broadwell? Message-ID: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding "nosmap" to the kernel command line in grub prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.) >From https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html, section 3.2: Note: In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor * Disable the Supervisor Mode Access Prevention (smap) kernel parameter * Reboot the RHEL 7 node before using GPFS Some observations worth noting: 1. We've been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs? 2. It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without "nosmap". Instead, an error message describing the fix would have made my day. 3. I'm going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. Thanks, Paul Sanchez -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jun 7 19:25:39 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 7 Jun 2016 11:25:39 -0700 Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 onHaswell/Broadwell? In-Reply-To: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> References: <565240ad49e6476da9c1d3d11312f88c@mbxpsc1.winmail.deshaw.com> Message-ID: Hi Paul, Yes, GPFS certainly needs to behave better in this situation. We are currently working on proper support for running on newer hardware that supports Superuser Mode Access Prevention (SMAP) instructions. I believe those are new to Broadwell CPUs, but there's some confusing info out there, I'm not positive what the deal is with Haswell. For the time being, booting with the "nosmap" kernel parameter is the workaround, but you're absolutely correct, the code needs to fail more gracefully when SMAP is enabled. We'll fix that. The current FAQ structure is, without question, suboptimal. We're looking for a better format to present this information, along the lines of more modern approaches like a structured Knowledge Base. The problem is recognized, on our end, but we've been having hard time making forward progress on this. yuri From: "Sanchez, Paul" To: "gpfsug main discussion list (gpfsug-discuss at spectrumscale.org)" , Date: 06/03/2016 06:38 AM Subject: [gpfsug-discuss] nosmap parameter for RHEL7 x86_64 on Haswell/Broadwell? Sent by: gpfsug-discuss-bounces at spectrumscale.org After some puzzling debugging on our new Broadwell servers, all of which slowly became brick-like upon after getting stuck starting GPFS, we discovered that this was already a known issue in the FAQ. Adding ?nosmap? to the kernel command line in grub prevents SMAP from seeing the kernel-userspace memory interactions of GPFS as a reason to slowly grind all cores to a standstill, apparently spinning on stuck locks(?). (Big thanks go to RedHat for turning us on to the answer when we opened a case.) From https://www.ibm.com/support/knowledgecenter/STXKQY/gpfsclustersfaq.html, section 3.2: Note: In order for IBM Spectrum Scale on RHEL 7 to run on the Haswell processor Disable the Supervisor Mode Access Prevention (smap) kernel parameter Reboot the RHEL 7 node before using GPFS Some observations worth noting: 1. We?ve been running for a year with Haswell processors and have hundreds of Haswell RHEL7 nodes which do not exhibit this problem. So maybe this only really affects Broadwell CPUs? 2. It would be very nice for SpectrumScale to take a peek at /proc/cpuinfo and /proc/cmdline before starting up, and refuse to break the host when it has affected processors and kernel without ?nosmap?. Instead, an error message describing the fix would have made my day. 3. I?m going to have to start using a script to diff the FAQ for these gotchas, unless anyone knows of a better way to subscribe just to updates to this doc. Thanks, Paul Sanchez _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 8 15:50:10 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 8 Jun 2016 14:50:10 +0000 Subject: [gpfsug-discuss] GPFS Tuning Wiki Page Message-ID: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> Hi All, https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Linux%20Tuning The above Wiki page describes itself as "GPFS network tuning parameters for large clusters on Linux.? How does GPFS multi-cluster play into this equation? For example, we?ve got a ~700 node traditional HPC cluster where these parameters would obviously come into play. But we?ve also got another 9 node GPFS cluster that is mainly accessed via CNFS and CIFS from labs around campus. However, it?s filesystem is also natively GPFS mounted on the HPC cluster via GPFS multi-cluster. Do I therefore need to apply the tuning referenced in the Wiki page above to the 9 node cluster? Thanks? ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Jun 8 21:31:04 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 8 Jun 2016 22:31:04 +0200 Subject: [gpfsug-discuss] GPFS Tuning Wiki Page In-Reply-To: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> References: <01177F16-FA40-41C1-A5CC-B78C396581B0@vanderbilt.edu> Message-ID: An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Thu Jun 9 16:21:07 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Thu, 9 Jun 2016 16:21:07 +0100 Subject: [gpfsug-discuss] Dry-Run functionality Message-ID: <57598963.3040706@pixitmedia.com> Hello all, I wonder if you could have your take on your ideal dry-run functionality. Post London User Group, we had much feedback regarding the need for dry-run functionality. Therefore we have implemented dry-run functionality for our GPFS/Scale Python API across all calls. API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development Pixit Media & ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Luke.Raimbach at crick.ac.uk Thu Jun 9 16:26:49 2016 From: Luke.Raimbach at crick.ac.uk (Luke Raimbach) Date: Thu, 9 Jun 2016 15:26:49 +0000 Subject: [gpfsug-discuss] Number of Inodes in chained AFM Caches Message-ID: Hi All, We have a situation where we are chaining AFM caches together: local update on the end of a read-only cache of a non GPFS NFS share. Essentially we have an intermediate read-only cache which can pull multiple data sources in quickly over the local network at the remote site, staging the data for WAN transfer into the ultimate destination which is a local update cache. In our first set of test runs, I am seeing a large discrepancy in the number of allocated inodes in the intermediate cache and the final local-update cache. We have sized the intermediate read-only caches to contain about 10% more "Maximum number of inodes" than the source NFS share (to accommodate any growth that happens while we transfer the data). We have also size the final local-update cache to have the same maximum number of inodes. The number of allocated inodes in the intermediate read-only cache shows the number of files in the source NFS share. However in the local-update shares, I always see 1500800 allocated and am concerned that the destination is not reading all the file-system metadata from the intermediate cache. Any ideas what could be going on here? Cheers, Luke. Luke Raimbach? Senior HPC Data and Storage Systems Engineer, The Francis Crick Institute, Gibbs Building, 215 Euston Road, London NW1 2BE. E: luke.raimbach at crick.ac.uk W: www.crick.ac.uk The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE. From Robert.Oesterlin at nuance.com Sun Jun 12 01:20:23 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Sun, 12 Jun 2016 00:20:23 +0000 Subject: [gpfsug-discuss] GPFSUG Meeting at Argonne 6/10 - Presentations Message-ID: <108228BB-1A66-4CE4-86D1-0CE9B3C8196F@nuance.com> The presentations from the user group meeting on June10th at Argronne labs are now available on the spectrumscale.org site here: http://www.spectrumscale.org/presentations/ Both my myself and Kristy Kallback-Rose thank the following who helped make it happen: Argonne Labs: Bill Allcock, Gordon McPheeters IBM team: Doris Conti, Doug O'flaherty, Ulf Troppens Along with all the presenters! We'll be following up with a more detailed report in the coming days, look for that on the spectrumscale.org site too. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid SSUG-USA Co-Principal -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Mon Jun 13 17:11:07 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 12:11:07 -0400 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> Message-ID: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From cphoffma at lanl.gov Mon Jun 13 17:19:02 2016 From: cphoffma at lanl.gov (Hoffman, Christopher P) Date: Mon, 13 Jun 2016 16:19:02 +0000 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca>, <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: Hi Jaime, What in particular would you like explained more? I'd be more than happy to discuss things further. Chris ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] Sent: Monday, June 13, 2016 10:11 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS on ZFS? I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From stijn.deweirdt at ugent.be Mon Jun 13 17:25:55 2016 From: stijn.deweirdt at ugent.be (Stijn De Weirdt) Date: Mon, 13 Jun 2016 18:25:55 +0200 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: <90312ed5-f263-a973-c768-28f535dc1316@ugent.be> hi chris, do you have any form of HA for the zfs blockdevices/jbod (eg when a nsd reboots/breaks/...)? or do you rely on replication within GPFS? stijn On 06/13/2016 06:19 PM, Hoffman, Christopher P wrote: > Hi Jaime, > > What in particular would you like explained more? I'd be more than happy to discuss things further. > > Chris > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] > Sent: Monday, June 13, 2016 10:11 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > > I just came across this presentation on "GPFS with underlying ZFS > block devices", by Christopher Hoffman, Los Alamos National Lab, > although some of the > implementation remains obscure. > > http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf > > It would be great to have more details, in particular the possibility > of straight use of GPFS on ZFS, instead of the 'archive' use case as > described on the presentation. > > Thanks > Jaime > > > > > Quoting "Jaime Pinto" : > >> Since we can not get GNR outside ESS/GSS appliances, is anybody using >> ZFS for software raid on commodity storage? >> >> Thanks >> Jaime >> >> > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > From pinto at scinet.utoronto.ca Mon Jun 13 17:26:27 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 12:26:27 -0400 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca>, <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> Hi Chris As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. How is that accomplished when you use ZFS for software raid? On page 4 I see this info, and I'm trying to interpret it: General Configuration ... * zvols * nsddevices ? echo "zdX generic? Thanks Jaime Quoting "Hoffman, Christopher P" : > Hi Jaime, > > What in particular would you like explained more? I'd be more than > happy to discuss things further. > > Chris > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] > Sent: Monday, June 13, 2016 10:11 > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > > I just came across this presentation on "GPFS with underlying ZFS > block devices", by Christopher Hoffman, Los Alamos National Lab, > although some of the > implementation remains obscure. > > http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf > > It would be great to have more details, in particular the possibility > of straight use of GPFS on ZFS, instead of the 'archive' use case as > described on the presentation. > > Thanks > Jaime > > > > > Quoting "Jaime Pinto" : > >> Since we can not get GNR outside ESS/GSS appliances, is anybody using >> ZFS for software raid on commodity storage? >> >> Thanks >> Jaime >> >> > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From bsallen at alcf.anl.gov Mon Jun 13 17:33:54 2016 From: bsallen at alcf.anl.gov (Allen, Benjamin S.) Date: Mon, 13 Jun 2016 16:33:54 +0000 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> Message-ID: <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Jaime, See https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm. An example I have for add /dev/nvme* devices: * GPFS doesn't know how that /dev/nvme* are valid block devices, use a user exit script to let it know about it cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices * Edit /var/mmfs/etc/nsddevices, and add to linux section: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) do echo $dev generic done fi * Copy edited nsddevices to the rest of the nodes at the same path for host in n01 n02 n03 n04; do scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices done Ben > On Jun 13, 2016, at 11:26 AM, Jaime Pinto wrote: > > Hi Chris > > As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. > > How is that accomplished when you use ZFS for software raid? > On page 4 I see this info, and I'm trying to interpret it: > > General Configuration > ... > * zvols > * nsddevices > - echo "zdX generic" > > > Thanks > Jaime > > Quoting "Hoffman, Christopher P" : > >> Hi Jaime, >> >> What in particular would you like explained more? I'd be more than happy to discuss things further. >> >> Chris >> ________________________________________ >> From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] >> Sent: Monday, June 13, 2016 10:11 >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >> >> I just came across this presentation on "GPFS with underlying ZFS >> block devices", by Christopher Hoffman, Los Alamos National Lab, >> although some of the >> implementation remains obscure. >> >> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >> >> It would be great to have more details, in particular the possibility >> of straight use of GPFS on ZFS, instead of the 'archive' use case as >> described on the presentation. >> >> Thanks >> Jaime >> >> >> >> >> Quoting "Jaime Pinto" : >> >>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>> ZFS for software raid on commodity storage? >>> >>> Thanks >>> Jaime >>> >>> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From makaplan at us.ibm.com Mon Jun 13 18:53:41 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 13 Jun 2016 13:53:41 -0400 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: How do you set the size of a ZFS file that is simulating a GPFS disk? How do "tell" GPFS about that? How efficient is this layering, compared to just giving GPFS direct access to the same kind of LUNs that ZFS is using? Hmmm... to partially answer my question, I do something similar, but strictly for testing non-performance critical GPFS functions. On any file system one can: dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a fake 3GB disk for GPFS Then use a GPFS nsd configuration record like this: %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra servers=bog-xxx Which starts out as sparse and the filesystem will dynamically "grow" as GPFS writes to it... But I have no idea how well this will work for a critical "production" system... tx, marc kaplan. From: "Allen, Benjamin S." To: gpfsug main discussion list Date: 06/13/2016 12:34 PM Subject: Re: [gpfsug-discuss] GPFS on ZFS? Sent by: gpfsug-discuss-bounces at spectrumscale.org Jaime, See https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm . An example I have for add /dev/nvme* devices: * GPFS doesn't know how that /dev/nvme* are valid block devices, use a user exit script to let it know about it cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices * Edit /var/mmfs/etc/nsddevices, and add to linux section: if [[ $osName = Linux ]] then : # Add function to discover disks in the Linux environment. for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) do echo $dev generic done fi * Copy edited nsddevices to the rest of the nodes at the same path for host in n01 n02 n03 n04; do scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices done Ben > On Jun 13, 2016, at 11:26 AM, Jaime Pinto wrote: > > Hi Chris > > As I understand, GPFS likes to 'see' the block devices, even on a hardware raid solution such at DDN's. > > How is that accomplished when you use ZFS for software raid? > On page 4 I see this info, and I'm trying to interpret it: > > General Configuration > ... > * zvols > * nsddevices > - echo "zdX generic" > > > Thanks > Jaime > > Quoting "Hoffman, Christopher P" : > >> Hi Jaime, >> >> What in particular would you like explained more? I'd be more than happy to discuss things further. >> >> Chris >> ________________________________________ >> From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] >> Sent: Monday, June 13, 2016 10:11 >> To: gpfsug main discussion list >> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >> >> I just came across this presentation on "GPFS with underlying ZFS >> block devices", by Christopher Hoffman, Los Alamos National Lab, >> although some of the >> implementation remains obscure. >> >> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >> >> It would be great to have more details, in particular the possibility >> of straight use of GPFS on ZFS, instead of the 'archive' use case as >> described on the presentation. >> >> Thanks >> Jaime >> >> >> >> >> Quoting "Jaime Pinto" : >> >>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>> ZFS for software raid on commodity storage? >>> >>> Thanks >>> Jaime >>> >>> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > > > > > > > ************************************ > TELL US ABOUT YOUR SUCCESS STORIES > http://www.scinethpc.ca/testimonials > ************************************ > --- > Jaime Pinto > SciNet HPC Consortium - Compute/Calcul Canada > www.scinet.utoronto.ca - www.computecanada.org > University of Toronto > 256 McCaul Street, Room 235 > Toronto, ON, M5T1W5 > P: 416-978-2755 > C: 416-505-1477 > > ---------------------------------------------------------------- > This message was sent using IMP at SciNet Consortium, University of Toronto. > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Mon Jun 13 19:02:51 2016 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Mon, 13 Jun 2016 14:02:51 -0400 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> As Marc, I also have questions related to performance. Assuming we let ZFS take care of the underlying software raid, what would be the difference between GPFS and Lustre for instance, for the "parallel serving" at scale part of the file system. What would keep GPFS from performing or functioning just as well? Thanks Jaime Quoting "Marc A Kaplan" : > How do you set the size of a ZFS file that is simulating a GPFS disk? How > do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct access > to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > > tx, marc kaplan. > > > > From: "Allen, Benjamin S." > To: gpfsug main discussion list > Date: 06/13/2016 12:34 PM > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Jaime, > > See > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > . An example I have for add /dev/nvme* devices: > > * GPFS doesn't know how that /dev/nvme* are valid block devices, use a > user exit script to let it know about it > > cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices > > * Edit /var/mmfs/etc/nsddevices, and add to linux section: > > if [[ $osName = Linux ]] > then > : # Add function to discover disks in the Linux environment. > for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) > do > echo $dev generic > done > fi > > * Copy edited nsddevices to the rest of the nodes at the same path > for host in n01 n02 n03 n04; do > scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices > done > > > Ben > >> On Jun 13, 2016, at 11:26 AM, Jaime Pinto > wrote: >> >> Hi Chris >> >> As I understand, GPFS likes to 'see' the block devices, even on a > hardware raid solution such as DDN's. >> >> How is that accomplished when you use ZFS for software raid? >> On page 4, I see this info, and I'm trying to interpret it: >> >> General Configuration >> ... >> * zvols >> * nsddevices >> - echo "zdX generic" >> >> >> Thanks >> Jaime >> >> Quoting "Hoffman, Christopher P" : >> >>> Hi Jaime, >>> >>> What in particular would you like explained more? I'd be more than > happy to discuss things further. >>> >>> Chris >>> ________________________________________ >>> From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] >>> Sent: Monday, June 13, 2016 10:11 >>> To: gpfsug main discussion list >>> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >>> >>> I just came across this presentation on "GPFS with underlying ZFS >>> block devices", by Christopher Hoffman, Los Alamos National Lab, >>> although some of the >>> implementation remains obscure. >>> >>> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >>> >>> It would be great to have more details, in particular the possibility >>> of straight use of GPFS on ZFS, instead of the 'archive' use case as >>> described on the presentation. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> Quoting "Jaime Pinto" : >>> >>>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>>> ZFS for software raid on commodity storage? >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> http://www.scinethpc.ca/testimonials >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.org >>> University of Toronto >>> 256 McCaul Street, Room 235 >>> Toronto, ON, M5T1W5 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of > Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of > Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. From cphoffma at lanl.gov Mon Jun 13 19:15:37 2016 From: cphoffma at lanl.gov (Hoffman, Christopher P) Date: Mon, 13 Jun 2016 18:15:37 +0000 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca><20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca><20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> , <20160613140251.164626kxtyddt64r@support.scinet.utoronto.ca> Message-ID: To specify the size of the disks GPFS uses one can use zvols. Then one can turn on the zfs setting sync=always to perform safe writes since I'm using SATA cards there is no BBU. In our testing, turning sync=on creates a 20%-30% decrease in overall throughput on writes. I do not have numbers of this setup vs hardware RAID6. Thanks, Chris ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto [pinto at scinet.utoronto.ca] Sent: Monday, June 13, 2016 12:02 To: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] GPFS on ZFS! ... ? As Marc, I also have questions related to performance. Assuming we let ZFS take care of the underlying software raid, what would be the difference between GPFS and Lustre for instance, for the "parallel serving" at scale part of the file system. What would keep GPFS from performing or functioning just as well? Thanks Jaime Quoting "Marc A Kaplan" : > How do you set the size of a ZFS file that is simulating a GPFS disk? How > do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct access > to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > > tx, marc kaplan. > > > > From: "Allen, Benjamin S." > To: gpfsug main discussion list > Date: 06/13/2016 12:34 PM > Subject: Re: [gpfsug-discuss] GPFS on ZFS? > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > > > Jaime, > > See > https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm > . An example I have for add /dev/nvme* devices: > > * GPFS doesn't know how that /dev/nvme* are valid block devices, use a > user exit script to let it know about it > > cp /usr/lpp/mmfs/samples/nsddevices.sample /var/mmfs/etc/nsddevices > > * Edit /var/mmfs/etc/nsddevices, and add to linux section: > > if [[ $osName = Linux ]] > then > : # Add function to discover disks in the Linux environment. > for dev in $( cat /proc/partitions | grep nvme | awk '{print $4}' ) > do > echo $dev generic > done > fi > > * Copy edited nsddevices to the rest of the nodes at the same path > for host in n01 n02 n03 n04; do > scp /var/mmfs/etc/nsddevices ${host}:/var/mmfs/etc/nsddevices > done > > > Ben > >> On Jun 13, 2016, at 11:26 AM, Jaime Pinto > wrote: >> >> Hi Chris >> >> As I understand, GPFS likes to 'see' the block devices, even on a > hardware raid solution such as DDN's. >> >> How is that accomplished when you use ZFS for software raid? >> On page 4, I see this info, and I'm trying to interpret it: >> >> General Configuration >> ... >> * zvols >> * nsddevices >> - echo "zdX generic" >> >> >> Thanks >> Jaime >> >> Quoting "Hoffman, Christopher P" : >> >>> Hi Jaime, >>> >>> What in particular would you like explained more? I'd be more than > happy to discuss things further. >>> >>> Chris >>> ________________________________________ >>> From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Jaime Pinto > [pinto at scinet.utoronto.ca] >>> Sent: Monday, June 13, 2016 10:11 >>> To: gpfsug main discussion list >>> Subject: Re: [gpfsug-discuss] GPFS on ZFS? >>> >>> I just came across this presentation on "GPFS with underlying ZFS >>> block devices", by Christopher Hoffman, Los Alamos National Lab, >>> although some of the >>> implementation remains obscure. >>> >>> http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf >>> >>> It would be great to have more details, in particular the possibility >>> of straight use of GPFS on ZFS, instead of the 'archive' use case as >>> described on the presentation. >>> >>> Thanks >>> Jaime >>> >>> >>> >>> >>> Quoting "Jaime Pinto" : >>> >>>> Since we can not get GNR outside ESS/GSS appliances, is anybody using >>>> ZFS for software raid on commodity storage? >>>> >>>> Thanks >>>> Jaime >>>> >>>> >>> >>> >>> >>> >>> ************************************ >>> TELL US ABOUT YOUR SUCCESS STORIES >>> http://www.scinethpc.ca/testimonials >>> ************************************ >>> --- >>> Jaime Pinto >>> SciNet HPC Consortium - Compute/Calcul Canada >>> www.scinet.utoronto.ca - www.computecanada.org >>> University of Toronto >>> 256 McCaul Street, Room 235 >>> Toronto, ON, M5T1W5 >>> P: 416-978-2755 >>> C: 416-505-1477 >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP at SciNet Consortium, University of > Toronto. >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >> >> >> >> >> >> >> ************************************ >> TELL US ABOUT YOUR SUCCESS STORIES >> http://www.scinethpc.ca/testimonials >> ************************************ >> --- >> Jaime Pinto >> SciNet HPC Consortium - Compute/Calcul Canada >> www.scinet.utoronto.ca - www.computecanada.org >> University of Toronto >> 256 McCaul Street, Room 235 >> Toronto, ON, M5T1W5 >> P: 416-978-2755 >> C: 416-505-1477 >> >> ---------------------------------------------------------------- >> This message was sent using IMP at SciNet Consortium, University of > Toronto. >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From jonathan at buzzard.me.uk Mon Jun 13 20:05:20 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Mon, 13 Jun 2016 20:05:20 +0100 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> Message-ID: <575F03F0.2020605@buzzard.me.uk> On 13/06/16 18:53, Marc A Kaplan wrote: > How do you set the size of a ZFS file that is simulating a GPFS disk? > How do "tell" GPFS about that? > > How efficient is this layering, compared to just giving GPFS direct > access to the same kind of LUNs that ZFS is using? > > Hmmm... to partially answer my question, I do something similar, but > strictly for testing non-performance critical GPFS functions. > On any file system one can: > > dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create a > fake 3GB disk for GPFS > > Then use a GPFS nsd configuration record like this: > > %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra > servers=bog-xxx > > Which starts out as sparse and the filesystem will dynamically "grow" as > GPFS writes to it... > > But I have no idea how well this will work for a critical "production" > system... > For "testing" purposes I just create a logical volume and map it through to my bunch of GPFS KVM instances as a disk. Works a treat and SSD's are silly money these days so for testing performance is just fine. There was a 960GB SanDisk on offer for 160GBP last month. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From laurence at qsplace.co.uk Mon Jun 13 20:12:03 2016 From: laurence at qsplace.co.uk (Laurence Horrocks-Barlow) Date: Mon, 13 Jun 2016 22:12:03 +0300 Subject: [gpfsug-discuss] GPFS on ZFS! ... ? In-Reply-To: <575F03F0.2020605@buzzard.me.uk> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> <20160613122627.16177059bagxk6gj@support.scinet.utoronto.ca> <95C7A0FB-D1B5-4D22-9F09-EF87D7E21CD9@alcf.anl.gov> <575F03F0.2020605@buzzard.me.uk> Message-ID: <9963179A-1D12-44BD-ABE7-5A607D0644D8@qsplace.co.uk> @JAB Same here passing the same LVM LV's through to multiple KVM instances works a treat for testing. -- Lauz On 13 June 2016 22:05:20 EEST, Jonathan Buzzard wrote: >On 13/06/16 18:53, Marc A Kaplan wrote: >> How do you set the size of a ZFS file that is simulating a GPFS disk? >> How do "tell" GPFS about that? >> >> How efficient is this layering, compared to just giving GPFS direct >> access to the same kind of LUNs that ZFS is using? >> >> Hmmm... to partially answer my question, I do something similar, but >> strictly for testing non-performance critical GPFS functions. >> On any file system one can: >> >> dd if=/dev/zero of=/fakedisks/d3 count=1 bs=1M seek=3000 # create >a >> fake 3GB disk for GPFS >> >> Then use a GPFS nsd configuration record like this: >> >> %nsd: nsd=d3 device=/fakedisks/d3 usage=dataOnly pool=xtra >> servers=bog-xxx >> >> Which starts out as sparse and the filesystem will dynamically "grow" >as >> GPFS writes to it... >> >> But I have no idea how well this will work for a critical >"production" >> system... >> > >For "testing" purposes I just create a logical volume and map it >through >to my bunch of GPFS KVM instances as a disk. Works a treat and SSD's >are >silly money these days so for testing performance is just fine. There >was a 960GB SanDisk on offer for 160GBP last month. > >JAB. > >-- >Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk >Fife, United Kingdom. >_______________________________________________ >gpfsug-discuss mailing list >gpfsug-discuss at spectrumscale.org >http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Tue Jun 14 21:50:08 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Tue, 14 Jun 2016 20:50:08 +0000 Subject: [gpfsug-discuss] QoS question Message-ID: Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From volobuev at us.ibm.com Tue Jun 14 22:05:53 2016 From: volobuev at us.ibm.com (Yuri L Volobuev) Date: Tue, 14 Jun 2016 14:05:53 -0700 Subject: [gpfsug-discuss] GPFS on ZFS? In-Reply-To: <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> References: <20160418095414.10636zytueeqmupy@support.scinet.utoronto.ca> <20160613121107.98347ddnaufjc6jf@support.scinet.utoronto.ca> Message-ID: GPFS proper (as opposed to GNR) isn't particularly picky about block devices. Any block device that GPFS can see, with help from an nsddevices user exit if necessary, is fair game, for those willing to blaze new trails. This applies to "real" devices, e.g. disk partitions or hardware RAID LUNs, and "virtual" ones, like software RAID devices. The device has to be capable to accepting IO requests of GPFS block size, but aside from that, Linux kernel does a pretty good job abstracting the realities of low-level implementation from the higher-level block device API. The basic problem with software RAID approaches is the lack of efficient HA. Since a given device is only visible to one node, if a node goes down, it takes the NSDs with it (as opposed to the more traditional twin-tailed disk model, when another NSD server can take over). So one would have to rely on GPFS data/metadata replication to get HA, and that is costly, in terms of disk utilization efficiency and data write cost. This is still an attractive model for some use cases, but it's not quite a one-to-one replacement for something like GNR for general use. yuri From: "Jaime Pinto" To: "gpfsug main discussion list" , Date: 06/13/2016 09:11 AM Subject: Re: [gpfsug-discuss] GPFS on ZFS? Sent by: gpfsug-discuss-bounces at spectrumscale.org I just came across this presentation on "GPFS with underlying ZFS block devices", by Christopher Hoffman, Los Alamos National Lab, although some of the implementation remains obscure. http://files.gpfsug.org/presentations/2016/anl-june/LANL_GPFS_ZFS.pdf It would be great to have more details, in particular the possibility of straight use of GPFS on ZFS, instead of the 'archive' use case as described on the presentation. Thanks Jaime Quoting "Jaime Pinto" : > Since we can not get GNR outside ESS/GSS appliances, is anybody using > ZFS for software raid on commodity storage? > > Thanks > Jaime > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.org University of Toronto 256 McCaul Street, Room 235 Toronto, ON, M5T1W5 P: 416-978-2755 C: 416-505-1477 ---------------------------------------------------------------- This message was sent using IMP at SciNet Consortium, University of Toronto. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Tue Jun 14 23:07:36 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Tue, 14 Jun 2016 18:07:36 -0400 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: Yes, in QOS for 4.2.0 there are some simple assumptions that may not make a lot of sense in some configurations, especially configurations with many (100s) of nodes mounting the same file system... You can try out what you suggested and in 4.2.0 I think it will pretty much work as you suggest -- essentially you are allocating 466 maintenance iops to every node, knowing that most of those nodes will not be using their allocation of IOPS. In later releases, you may find that we will address some of these kinds of quirks in QOS. QOS is a new feature for GPFS, and I don't think you'll find anything like it in any commercial file system offering. (Correct me with example(s) if I am wrong on this point.) So think of it as "release 1.0" (of QOS) and let us know how well it works for you and how it might be improved.... --marc of Spectrum(GP)FS From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2016 04:50 PM Subject: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Wed Jun 15 06:54:54 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Wed, 15 Jun 2016 05:54:54 +0000 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: XFS on Irix had a feature similar to QoS, called GRIO (guaranteed rate I/O), where applications could reserve a given bandwidth. http://www.sgistuff.net/software/irixintro/documents/xfs-whitepaper.html Sounds somewhat similar to QoS, but focused on giving applications guaranteed bandwidth, not iops. -jf ons. 15. jun. 2016 kl. 00.08 skrev Marc A Kaplan : > Yes, in QOS for 4.2.0 there are some simple assumptions that may not make > a lot of sense in some configurations, especially configurations with many > (100s) of nodes mounting the same file system... You can try out what > you suggested and in 4.2.0 I think it will pretty much work as you suggest > -- essentially you are allocating 466 maintenance iops to every node, > knowing that most of those nodes will not be using their allocation of IOPS. > > In later releases, you may find that we will address some of these kinds > of quirks in QOS. > > QOS is a new feature for GPFS, and I don't think you'll find anything like > it in any commercial file system offering. (Correct me with example(s) if > I am wrong on this point.) > So think of it as "release 1.0" (of QOS) and let us know how well it works > for you and how it might be improved.... > > --marc of Spectrum(GP)FS > > > > From: "Buterbaugh, Kevin L" > To: gpfsug main discussion list > Date: 06/14/2016 04:50 PM > Subject: [gpfsug-discuss] QoS question > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Hi All, > > We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to > dive into my first attempts at using the new QoS features. I want to make > sure I am understanding the documentation: > > "The IOPS values that you set in an mmchqos command apply to all I/O > operations that are issued by all the nodes that have the specified file > system mounted. You should adjust your allocations of IOPS accordingly. > > For example, if you 600 IOPS to the maintenance class, and there are six > nodes that have the file system mounted, then QoS allocates 100 IOPS to the > maintenance class of each node. If you then run maintenance commands that > affect only three of the nodes, the commands runs with an actual allocation > of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect > three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate > 1200 IOPS to the maintenanceclass. " > > We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret > the above assuming that I have determined that I want to allow 7,000 IOPs ? > please correct me if I?m wrong... > > 7,000 IOPs / 700 nodes would be 10 IOPs per node. > > But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that > are going to be doing the maintenance (a restripe, for example), so 7,000 / > 15 = 466.67. > > 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 > IOPs to the maintenance class? > > Thanks in advance? > > Kevin > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > > *Kevin.Buterbaugh at vanderbilt.edu* - > (615)875-9633 > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From TROPPENS at de.ibm.com Wed Jun 15 12:03:10 2016 From: TROPPENS at de.ibm.com (Ulf Troppens) Date: Wed, 15 Jun 2016 13:03:10 +0200 Subject: [gpfsug-discuss] ISC U/G Agenda - Next Monday In-Reply-To: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com> References: <201605131502.u4DF23Bi020162@d03av05.boulder.ibm.com> Message-ID: <201606151103.u5FAwsYp073979@mx0a-001b2d01.pphosted.com> Hello Everybody, here is a reminder for the User Group meeting at ISC in Frankfurt next Monday. Organized by IBM ... but same spirit as the meetings in London, New York and Chicago. See here for updated agenda and registration: http://www-05.ibm.com/de/events/isc16/ Looking forward to see many of you next Monday. Best regards, Ulf. -- IBM Spectrum Scale Development - Client Engagements & Solutions Delivery Consulting IT Specialist Author "Storage Networks Explained" IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Douglas O'flaherty" To: gpfsug-discuss at spectrumscale.org Date: 13.05.2016 17:02 Subject: [gpfsug-discuss] ISC U/G Agenda Sent by: gpfsug-discuss-bounces at spectrumscale.org Greetings: IBM is happy to announce the agenda for the joint IBM Spectrum Scale and Platform Computing User Group Agenda at ISC. We will finish on time to attend the opening reception. As with other user group meetings, the agenda includes user stories, updates on IBM Spectrum Scale & Platform LSF, and access to IBM experts and your peers. Please join us! To attend, please email Fabian.Beckmann at de.ibm.com so we can have an accurate count of attendees. Monday June 20, 2016 - 14:30-18:00 - Conference Room Konstant 14:30-14:40 [10 min] Welcome (Douglas o'Flaherty, IBM) 14:40-15:00 [20 min] Ten Reasons to Upgrade from GPFS 3.4 to Spectrum Scale 4.2 (Olaf Weiser, IBM) 15:00-15:30 [30 min] Shared Storage with in-memory latency: EMC DSSD D5 and IBM Spectrum Scale (Stefan Radtke, EMC) 15:30-16:00 [30 min] Workload scheduling and data management in a private cloud (Uwe Sommer, Airbus) 16:00-16:30 [30 min] Spectrum Scale site report (To be confirmed by customer) 16:30-17:00 [30 min] What's new in Platform LSF 10.1 & storage integration (Bill McMillan, IBM) 17:00-17:30 [30 min] What's new in Spectrum Scale 4.2.1 (Mathias Dietz, IBM) 17:30-18:00 [30 min] CORAL enhancements for Spectrum Scale (Sven Oehme, IBM) Looking forward to seeing you there! doug PS: IBMers can register their clients at this IBM link: https://w3-9033.ibm.com/events/ast/schedule/16isc.nsf _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From makaplan at us.ibm.com Wed Jun 15 19:24:07 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Wed, 15 Jun 2016 14:24:07 -0400 Subject: [gpfsug-discuss] QoS question In-Reply-To: References: Message-ID: Thanks for that reference. Further research indicates that GRIO on XFS required Irix/SGI hardware features and GRIO is not in any recent implementations of XFS. Functionally GRIO, when it worked was kind of a logical complement or opposite of GPFS/QOS. GRIO was to be used to reserve bandwidth for critical applications; whereas GPFS 4.2/QOS primary purpose to restrict the IO bandwidth consumed by not-performance critical applications. Of course if you restrict set X to use no more than B, than you have effectively reserved TotalAvailable-B for the set ~X. It may interest you to know that the primary GPFS/QOS enforcement mechanisms are based on an implementation of Token Bucket ( wikipedia.org/wiki/Token_bucket , US Patent 9178827 ) --marc From: Jan-Frode Myklebust To: gpfsug main discussion list Date: 06/15/2016 01:55 AM Subject: Re: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org XFS on Irix had a feature similar to QoS, called GRIO (guaranteed rate I/O), where applications could reserve a given bandwidth. http://www.sgistuff.net/software/irixintro/documents/xfs-whitepaper.html Sounds somewhat similar to QoS, but focused on giving applications guaranteed bandwidth, not iops. -jf ons. 15. jun. 2016 kl. 00.08 skrev Marc A Kaplan : Yes, in QOS for 4.2.0 there are some simple assumptions that may not make a lot of sense in some configurations, especially configurations with many (100s) of nodes mounting the same file system... You can try out what you suggested and in 4.2.0 I think it will pretty much work as you suggest -- essentially you are allocating 466 maintenance iops to every node, knowing that most of those nodes will not be using their allocation of IOPS. In later releases, you may find that we will address some of these kinds of quirks in QOS. QOS is a new feature for GPFS, and I don't think you'll find anything like it in any commercial file system offering. (Correct me with example(s) if I am wrong on this point.) So think of it as "release 1.0" (of QOS) and let us know how well it works for you and how it might be improved.... --marc of Spectrum(GP)FS From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/14/2016 04:50 PM Subject: [gpfsug-discuss] QoS question Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, We have recently upgraded to GPFS 4.2.0-3 and so I am getting ready to dive into my first attempts at using the new QoS features. I want to make sure I am understanding the documentation: "The IOPS values that you set in an mmchqos command apply to all I/O operations that are issued by all the nodes that have the specified file system mounted. You should adjust your allocations of IOPS accordingly. For example, if you 600 IOPS to the maintenance class, and there are six nodes that have the file system mounted, then QoS allocates 100 IOPS to the maintenance class of each node. If you then run maintenance commands that affect only three of the nodes, the commands runs with an actual allocation of 300 IOPS, or 100 IOPS per node. To run maintenance commands that affect three nodes with an actual allotment of 600 IOPS, or 200 per node, allocate 1200 IOPS to the maintenanceclass. " We have a ~700 node cluster with 15 NSD servers. Here?s how I interpret the above assuming that I have determined that I want to allow 7,000 IOPs ? please correct me if I?m wrong... 7,000 IOPs / 700 nodes would be 10 IOPs per node. But I want those 7,000 IOPs to be divided amongst my 15 NSD servers that are going to be doing the maintenance (a restripe, for example), so 7,000 / 15 = 466.67. 466.67 * 700 (nodes in cluster) = 326,666.67. So I would allocate 326,666 IOPs to the maintenance class? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu- (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 15:29:20 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 14:29:20 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? Message-ID: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 15:35:10 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 14:35:10 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: Message-ID: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> Hi All, And, of course, within 5 minutes of sending that e-mail I found the ?mmlsattr? command, which does what I need and which didn?t show up in my Google search results. I apologize for the noise? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamiedavis at us.ibm.com Fri Jun 17 16:04:17 2016 From: jamiedavis at us.ibm.com (James Davis) Date: Fri, 17 Jun 2016 15:04:17 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> References: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu>, Message-ID: An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 16:18:48 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 15:18:48 +0000 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: References: Message-ID: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Hi yet again all, Well, this has turned out to be an enlightening and surprising morning in GPFS land? What prompted my question below is this ? I am looking to use the new QoS features in GPFS 4.2. I have QoS enabled and am trying to get a baseline of IOPs so that I can determine how much I want to assign to the maintenance class (currently both maintenance and other are set to unlimited). To do this, I fired off a bonnie++ test from each of my NSD servers. The filesystem in question has two storage pools, the system pool and the capacity pool. The system pool is comprised of a couple of metadata only disks (SSD-based RAID 1 mirrors) and several data only disks (spinning HD-based RAID 6), while the capacity pool is comprised exclusively of data only disks (RAID 6). When the bonnie++?s were creating, reading, and rewriting the big file they create I was quite surprised to see mmlsqos show higher IOP?s on the capacity pool than the system pool by a factor of 10! As I was expecting those files to be being written to the system pool, this was quite surprising to me. Once I found the mmlsattr command, I ran it on one of the files being created and saw that it was indeed assigned to the capacity pool. The bonnie++?s finished before I could check the other files. I don?t have any file placement policies in effect for this filesystem, only file migration policies (each weekend any files in the system pool with an atime > 60 days get moved to the capacity pool and any files in the capacity pool with an atime < 60 days get moved to the system pool). In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.? This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don?t remember for sure) later. I was under the obviously mistaken impression that the ?first? data storage pool was the system pool, but that is clearly not correct. So my first question is, what is the definition of ?the first storage pool?? and my second question is, can the documentation be updated with the answer to my first question since it?s clearly ambiguous as written now? Thanks? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 16:21:48 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 15:21:48 +0000 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: <7E5115DC-C2D3-4883-93A7-690D824A8FC1@vanderbilt.edu> Message-ID: <9C2AC549-B85E-40EE-92CA-98983A4B3D85@vanderbilt.edu> Hi Jamie, The much longer e-mail I just sent to the list will explain the background behind all this, but to answer your question I actually only needed to know which storage pool the file resided in. Obviously, if I know which NSDs it resides on that would tell me the storage pool. Thanks? Kevin On Jun 17, 2016, at 10:04 AM, James Davis > wrote: Hi Kevin, It looks like you can use mmlsattr to query the storage pool the file resides on. Is that all you needed to know, or did you need the specific disks on which it resides? If you're looking for the specific disks, how are you using the mmlsattr command to accomplish that? Jamie Jamie Davis GPFS Functional Verification Test (FVT) jamiedavis at us.ibm.com ----- Original message ----- From: "Buterbaugh, Kevin L" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: Re: [gpfsug-discuss] Can you query which NSDs a given file resides on? Date: Fri, Jun 17, 2016 10:36 AM Hi All, And, of course, within 5 minutes of sending that e-mail I found the ?mmlsattr? command, which does what I need and which didn?t show up in my Google search results. I apologize for the noise? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L > wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 17 16:22:44 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 11:22:44 -0400 Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? In-Reply-To: References: Message-ID: Hint: tsdbfs More hints: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014215697 If you mess up, you didn't hear this from me ;-) From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/17/2016 10:30 AM Subject: [gpfsug-discuss] Can you query which NSDs a given file resides on? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jamiedavis at us.ibm.com Fri Jun 17 16:50:26 2016 From: jamiedavis at us.ibm.com (James Davis) Date: Fri, 17 Jun 2016 15:50:26 +0000 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu>, Message-ID: An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Fri Jun 17 16:55:21 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Fri, 17 Jun 2016 16:55:21 +0100 Subject: [gpfsug-discuss] Initial file placement In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: <1466178921.3773.107.camel@buzzard.phy.strath.ac.uk> On Fri, 2016-06-17 at 15:18 +0000, Buterbaugh, Kevin L wrote: > Hi yet again all, > > > Well, this has turned out to be an enlightening and surprising morning > in GPFS land? > > > What prompted my question below is this ? I am looking to use the new > QoS features in GPFS 4.2. I have QoS enabled and am trying to get a > baseline of IOPs so that I can determine how much I want to assign to > the maintenance class (currently both maintenance and other are set to > unlimited). To do this, I fired off a bonnie++ test from each of my > NSD servers. > > > The filesystem in question has two storage pools, the system pool and > the capacity pool. The system pool is comprised of a couple of > metadata only disks (SSD-based RAID 1 mirrors) and several data only > disks (spinning HD-based RAID 6), while the capacity pool is comprised > exclusively of data only disks (RAID 6). > My preference in these scenarios is to have the system pool for metadata only. Then have a "fast" pool and a "slow" pool. Pool names more than 7 characters is a recipe for backup headaches with TSM at least, so "fast" and "slow" are deliberately chosen as names :) That way you can tweak block sizes more optimally. > > When the bonnie++?s were creating, reading, and rewriting the big file > they create I was quite surprised to see mmlsqos show higher IOP?s on > the capacity pool than the system pool by a factor of 10! As I was > expecting those files to be being written to the system pool, this was > quite surprising to me. Once I found the mmlsattr command, I ran it > on one of the files being created and saw that it was indeed assigned > to the capacity pool. The bonnie++?s finished before I could check > the other files. > > > I don?t have any file placement policies in effect for this > filesystem, only file migration policies (each weekend any files in > the system pool with an atime > 60 days get moved to the capacity pool > and any files in the capacity pool with an atime < 60 days get moved > to the system pool). Note due to the way applications handle saving files by creating a new file, moving files from slow to fast disk is mostly a waste of effort in my experience. By the time you do that, 99% of the time the user has finished with the file anyway. When they haven't generally the slow disk has gazillions of free IOPs anyway. > > In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS > file system does not have a placement policy installed, all the data > is stored in the first data storage pool.? > My understanding is that no placement policy and it can go anywhere, where anywhere is the next available NSD with free space. I found that you needed to do something like this at the end of your policy file for the default placement. /* move files over 1GB onto nearline storage and keep them there */ RULE 'big' MIGRATE FROM POOL 'system' TO POOL 'nearline' WHERE FILE_SIZE > 1073741824 /* migrate old files to the RAID6 disks to keep the RAID1 disks free */ RULE 'ilm' MIGRATE FROM POOL 'system' THRESHOLD(90,70) WEIGHT(weighting) TO POOL 'slow' /* by default new files to the fast RAID1 disk unless full, then to RAID6 */ RULE 'new' SET POOL 'fast' LIMIT(95) RULE 'spillover' SET POOL 'slow' You have to stop allocation to the fast disk from ever approaching 100%. Otherwise a you get disk full errors on new file writes. Imagine you create a new file when there is 1KB of space free in the fast pool, then try and write 10KB of data; it will fail with an error. So first rule forces large files to the slow pool anyway (you can tweak this size to suite your environment) but it must be significantly less than the the space free when you hit 95% full on the fast disk. You then need to have a rule to kick the tiering migration in lower than when you start allocating to the slow pool. Finally you allocation new files to the fast disk till it gets quite full, before spilling over to the slow pool (which one presumes has loads of free capacity) if the fast pool fills up. Not shown you ideally need to kick in tiering migration every night so that it does not run during the day when users might notice the performance degradation. Ideally you want to have enough free space come the morning in your fast disk pool that all new files can be created there during the day. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From makaplan at us.ibm.com Fri Jun 17 16:56:59 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 11:56:59 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (doh!) In-Reply-To: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: Thanks for asking! Prior to Release 4.2 system pool was the default pool for storing file data and you had to write at least one policy SET POOL rule to make use of any NSDs that you assigned to some other pool. Now if a file system is formatted or upgraded to level 4.2 or higher, and there is a data pool defined, and there are no SET POOL policy rules, then the "first" such data pool becomes the default storage pool for file data. "metadata" is always stored in system pool. So how is the "first" data pool determined? It's usually the first data pool added to the file system. For many customers and installations there is only one such pool, so no problem - and this is exactly what they wanted all along (since we introduced "pools" in release 3.1) We received complaints over the years along the lines: "Heh! Why the heck do you think I made a data pool? I don't want to know about silly SET POOL rules and yet another mm command (mmchpolicy)! Just do it!" Well in 4.2 we did. If you look into the /var/adm/ras/mmfs.log.* file(s) you will see that during mmmount 4.2 will tell you.... Fri Jun 17 09:31:26.637 2016: [I] Command: mount yy Fri Jun 17 09:31:27.625 2016: [I] Loaded policy 'for file system yy': Parsed 4 policy rules. Fri Jun 17 09:31:27.626 2016: Policy has no storage pool placement rules. Fri Jun 17 09:31:27.627 2016: [I] Data will be stored in pool 'xtra'. Notice we DID NOT change the behavior for file systems at level 4.1 or prior, even when you upgrade the software. But when you upgrade the file system to 4.2 (for example to use QOS) ... From: "Buterbaugh, Kevin L" To: gpfsug main discussion list Date: 06/17/2016 11:19 AM Subject: [gpfsug-discuss] Initial file placement Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi yet again all, Well, this has turned out to be an enlightening and surprising morning in GPFS land? What prompted my question below is this ? I am looking to use the new QoS features in GPFS 4.2. I have QoS enabled and am trying to get a baseline of IOPs so that I can determine how much I want to assign to the maintenance class (currently both maintenance and other are set to unlimited). To do this, I fired off a bonnie++ test from each of my NSD servers. The filesystem in question has two storage pools, the system pool and the capacity pool. The system pool is comprised of a couple of metadata only disks (SSD-based RAID 1 mirrors) and several data only disks (spinning HD-based RAID 6), while the capacity pool is comprised exclusively of data only disks (RAID 6). When the bonnie++?s were creating, reading, and rewriting the big file they create I was quite surprised to see mmlsqos show higher IOP?s on the capacity pool than the system pool by a factor of 10! As I was expecting those files to be being written to the system pool, this was quite surprising to me. Once I found the mmlsattr command, I ran it on one of the files being created and saw that it was indeed assigned to the capacity pool. The bonnie++?s finished before I could check the other files. I don?t have any file placement policies in effect for this filesystem, only file migration policies (each weekend any files in the system pool with an atime > 60 days get moved to the capacity pool and any files in the capacity pool with an atime < 60 days get moved to the system pool). In the GPFS 4.2 Advanced Administration Guide, it states, ?If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.? This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don?t remember for sure) later. I was under the obviously mistaken impression that the ?first? data storage pool was the system pool, but that is clearly not correct. So my first question is, what is the definition of ?the first storage pool?? and my second question is, can the documentation be updated with the answer to my first question since it?s clearly ambiguous as written now? Thanks? Kevin On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L < Kevin.Buterbaugh at Vanderbilt.Edu> wrote: Hi All, I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance? Kevin ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Fri Jun 17 17:10:55 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 12:10:55 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (doh!) (DOH!) 4.1.1 In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: OOPS... The story I gave was almost correct. The cut over was at 4.1.1 (after 4.1 before 4.2) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 21994 bytes Desc: not available URL: From makaplan at us.ibm.com Fri Jun 17 17:20:22 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Fri, 17 Jun 2016 12:20:22 -0400 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (DOC!) In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: (Doc?... I ain't about to read a manual! But just to cover our bases we did stick it into the 4.1.1 pubs) Policy improvements: This release includes the following policy improvements: ... (summary of changes, page xv) ... Implicit SET POOL 'first-data-pool' rule For file systems that are at or have been upgraded to 4.1.1, the system recognizes that, even if no policy rules have been installed to a file system by mmchpolicy , data files should be stored in a non-system pool if available (rather than in the system pool, which is the default for earlier releases). For more information, see the following: Information Lifecycle Management chapter in the IBM Spectrum Scale: Advanced Administration Guide ... ( mmchpolicy man page 298) ... For file systems that are at or have been upgraded to 4.1.1 or later: If there are no SET POOL policy rules installed to a file system by mmchpolicy, the system acts as if the single rule SET POOL 'first-data-pool' is in effect, where first-data-pool is the firstmost non-system pool that is available for file data storage, if such a non-system pool is available. (?Firstmost? is the first according to an internal index of all pools.) However, if there are no policy rules installed and there is no non-system pool, the system acts as if SET POOL 'system' is in effect. This change applies only to file systems that were created at or upgraded to 4.1.1. Until a file system is upgraded, if no SET POOL rules are present (set by mmchpolicy) for the file system, all data will be stored in the 'system' pool. For information on GPFS policies, see the IBM Spectrum Scale: Advanced Administration Guide. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Kevin.Buterbaugh at Vanderbilt.Edu Fri Jun 17 17:33:39 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Fri, 17 Jun 2016 16:33:39 +0000 Subject: [gpfsug-discuss] Initial file placement - first storage pool is now used for data storage (DOC!) In-Reply-To: References: <37FC5AC9-361F-42CA-96B9-138EDC468217@vanderbilt.edu> Message-ID: <78BBA4B7-505F-4D21-BA14-BC430D93F993@vanderbilt.edu> Hi Marc, Read every last one of the thousands of pages of documentation that comprises the GPFS documentation set before doing any GPFS upgrades ? yes, I apparently *do* need to do that! ;-) The paragraph that I find ambiguous can be found on page 20 of the GPFS 4.2 Advanced Administration Guide (3rd paragraph from bottom): The placement policy defining the initial placement of newly created files and the rules for placement of * | restored data must be installed into GPFS with the mmchpolicy command. If a GPFS file system does not * | have a placement policy installed, all the data is stored in the first data storage pool. Only one placement policy can be installed at a time. If you switch from one placement policy to another, or make changes to a placement policy, that action has no effect on existing files. However, newly created files are always placed according to the currently installed placement policy. In my opinion, it should at least contain a pointer the pages you reference below. Preferentially, it would contain the definition of ?first data storage pool?. Thanks? Kevin On Jun 17, 2016, at 11:20 AM, Marc A Kaplan > wrote: (Doc?... I ain't about to read a manual! But just to cover our bases we did stick it into the 4.1.1 pubs) Policy improvements: This release includes the following policy improvements: ... (summary of changes, page xv) ... Implicit SET POOL 'first-data-pool' rule For file systems that are at or have been upgraded to 4.1.1, the system recognizes that, even if no policy rules have been installed to a file system by mmchpolicy, data files should be stored in a non-system pool if available (rather than in the system pool, which is the default for earlier releases). For more information, see the following: Information Lifecycle Management chapter in the IBM Spectrum Scale: Advanced Administration Guide ... ( mmchpolicy man page 298) ... For file systems that are at or have been upgraded to 4.1.1 or later: If there are no SET POOL policy rules installed to a file system by mmchpolicy, the system acts as if the single rule SET POOL 'first-data-pool' is in effect, where first-data-pool is the firstmost non-system pool that is available for file data storage, if such a non-system pool is available. (?Firstmost? is the first according to an internal index of all pools.) However, if there are no policy rules installed and there is no non-system pool, the system acts as if SET POOL 'system' is in effect. This change applies only to file systems that were created at or upgraded to 4.1.1. Until a file system is upgraded, if no SET POOL rules are present (set by mmchpolicy) for the file system, all data will be stored in the 'system' pool. For information on GPFS policies, see the IBM Spectrum Scale: Advanced Administration Guide. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 14:45:34 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 13:45:34 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions Message-ID: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don't know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set "force user = root" all the snapshots are available. I think the answer is SAMBA, right? We're running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 14:49:36 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 13:49:36 +0000 Subject: [gpfsug-discuss] Dry-Run functionality In-Reply-To: <57598963.3040706@pixitmedia.com> References: <57598963.3040706@pixitmedia.com> Message-ID: IMO I?d want to see an explanation of what will happen in plain English. ?This will create a policy that moves any file bigger than 10MB and older than 30 days to storage pool XYZ?. You might be able to tell I?ve never used GPFS policies except for initial placement so the above example might not be possible anyway ? I?m very used to the Powershell switch ?-whatif? which basically said ?I am going to do the following:? Cheers Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Jez Tucker Sent: 09 June 2016 16:21 To: gpfsug main discussion list Subject: [gpfsug-discuss] Dry-Run functionality Hello all, I wonder if you could have your take on your ideal dry-run functionality. Post London User Group, we had much feedback regarding the need for dry-run functionality. Therefore we have implemented dry-run functionality for our GPFS/Scale Python API across all calls. API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development Pixit Media & ArcaStream www.pixitmedia.com www.arcastream.com [Image removed by sender.] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD000.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD000.jpg URL: From Kevin.Buterbaugh at Vanderbilt.Edu Mon Jun 20 15:56:27 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Mon, 20 Jun 2016 14:56:27 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions In-Reply-To: References: Message-ID: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> Hi Richard, I can?t answer your question but I can tell you that we have experienced either the exact same thing you are or something very similar. It occurred for us after upgrading from GPFS 3.5 to 4.1.0.8 and it persists even after upgraded to GPFS 4.2.0.3 and the very latest sernet-samba. And to be clear, when we upgraded from GPFS 3.5 to 4.1 we did *not* upgrade SAMBA versions at that time. Therefore, I believe that something changed in GPFS. That doesn?t mean it?s GPFS? fault, of course. SAMBA may have been relying on a bugundocumented feature in GPFS that IBM fixed for all I know, and I?m obviously speculating here. The problem we see is that the .snapshots directory in each folder can be cd?d to but is empty. The snapshots are all there, however, if you: cd //.snapshots//rest/of/path/to/folder/in/question This obviously prevents users from being able to do their own recovery of files unless you do something like what you describe, which we are unwilling to do for security reasons. We have a ticket open with DDN? Kevin On Jun 20, 2016, at 8:45 AM, Sobey, Richard A > wrote: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don?t know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set ?force user = root? all the snapshots are available. I think the answer is SAMBA, right? We?re running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From r.sobey at imperial.ac.uk Mon Jun 20 16:03:28 2016 From: r.sobey at imperial.ac.uk (Sobey, Richard A) Date: Mon, 20 Jun 2016 15:03:28 +0000 Subject: [gpfsug-discuss] Snapshots / Windows previous versions In-Reply-To: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> References: <444210FB-0E5A-4103-88E6-5079CCD9E7D0@vanderbilt.edu> Message-ID: Thanks Kevin. We are upgrading to GPFS 4.2 and CES in a few weeks but our customers have come to like previous versions and indeed it is sort of a selling point for us. Samba is the only thing we?ve changed recently after the badlock debacle so I?m tempted to blame that, but who knows. If (when) I find out I?ll let everyone know. Richard From: gpfsug-discuss-bounces at spectrumscale.org [mailto:gpfsug-discuss-bounces at spectrumscale.org] On Behalf Of Buterbaugh, Kevin L Sent: 20 June 2016 15:56 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Snapshots / Windows previous versions Hi Richard, I can?t answer your question but I can tell you that we have experienced either the exact same thing you are or something very similar. It occurred for us after upgrading from GPFS 3.5 to 4.1.0.8 and it persists even after upgraded to GPFS 4.2.0.3 and the very latest sernet-samba. And to be clear, when we upgraded from GPFS 3.5 to 4.1 we did *not* upgrade SAMBA versions at that time. Therefore, I believe that something changed in GPFS. That doesn?t mean it?s GPFS? fault, of course. SAMBA may have been relying on a bugundocumented feature in GPFS that IBM fixed for all I know, and I?m obviously speculating here. The problem we see is that the .snapshots directory in each folder can be cd?d to but is empty. The snapshots are all there, however, if you: cd //.snapshots//rest/of/path/to/folder/in/question This obviously prevents users from being able to do their own recovery of files unless you do something like what you describe, which we are unwilling to do for security reasons. We have a ticket open with DDN? Kevin On Jun 20, 2016, at 8:45 AM, Sobey, Richard A > wrote: Hi all Can someone clarify if the ability for Windows to view snapshots as Previous Versions is exposed by SAMBA or GPFS? Basically, if suddenly my users cannot restore files from snapshots over a CIFS share, where should I be looking? I don?t know when this problem occurred, but within the last few weeks certainly our users with full control over their data now see no previous versions available, but if we export their fileset and set ?force user = root? all the snapshots are available. I think the answer is SAMBA, right? We?re running GPFS 3.5 and sernet-samba 4.2.9. Many thanks Richard _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 20 16:03:50 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Jun 2016 11:03:50 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 20 16:03:50 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 20 Jun 2016 11:03:50 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Mon Jun 20 16:10:13 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Mon, 20 Jun 2016 15:10:13 +0000 Subject: [gpfsug-discuss] ESS GL6 Message-ID: Couple of questions regarding Spectrum Scale 4.2 and ESS. We recently got our ESS delivered and are putting it in production this week. Previous to ESS we ran GPFS 3.5 and IBM DCS3700 storage arrays. My question about ESS and Spectrum Scale has to do with querying available free space and adding capacity to existing file system. In the old days of GPFS 3.5 I would create LUNs on 3700, zone them to appropriate hosts, and then see them as multipath devices on NSD servers. After that, I would create NSD disks and add them to the filesystem. With the ESS, however, I don't think the process is quite the same. IBM tech that was here installing the system has created all the "LUNs" or the equivalent in the ESS system. How do you I query what space is available to add to the existing filesystems, and then how do you actually add space? I am reading ESS RedBook but the answers are not obvious. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Mon Jun 20 16:25:31 2016 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Mon, 20 Jun 2016 17:25:31 +0200 Subject: [gpfsug-discuss] ESS GL6 In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Jun 20 17:29:15 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 20 Jun 2016 16:29:15 +0000 Subject: [gpfsug-discuss] ESS GL6 In-Reply-To: References: Message-ID: "mmlsrecoverygroup $name -L" will tell you how much raw capacity is left in a recoverygroup. You will then need to create a vdisk stanza file where you specify the capacity, blocksize, raidcode, etc. for the vdisk you want. (check "man mmcrvdisk") Then "mmcrvdisk stanzafile" to create the vdisk, and "mmcrnsd stanzafile" to create the NSDs. From then on it's standard gpfs. -jf man. 20. jun. 2016 kl. 17.25 skrev Olaf Weiser : > Hi Damir, > mmlsrecovergroup --> will show your RG > > mmlsrecoverygroup RG -L .. will provide capacity information > > or .. you can use the GUI > > with ESS / GNR , there's no need any more to create more than one > vdisk(=nsd) per RG for a pool > > a practical approach/example for you > so a file system consists of > 1 vdisk(NSD) for MetaData , RAID: 4WR , BS 1M in RG"left" > 1 vdisk(NSD) for MetaData , Raid : 4WR, BS 1M in RG "right" > 1 vdisk (NSD) for data , 8+3p , BS 1...16M .. depends on your > data/workload in RG "left" > 1 vdisk (NSD) for data , 8+3p , BS 1...16M .. depends on your > data/workload in RG "right" > > so 4 NSDs to provide everything you need to serve a file system .. > > > the size of the vdisks can be up to half of the capacity of your RG > > please note: if you come from an existing environment , and the file > system should be migrated to ESS (online) , you might hit some limitations > like > - blocksize (can not be changed) > - disk size.. depending on the existing storage pools/disk sizes.... > > > have fun > cheers > > Mit freundlichen Gr??en / Kind regards > > > Olaf Weiser > > EMEA Storage Competence Center Mainz, German / IBM Systems, Storage > Platform, > > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland > IBM Allee 1 > 71139 Ehningen > Phone: +49-170-579-44-66 > E-Mail: olaf.weiser at de.ibm.com > > ------------------------------------------------------------------------------------------------------------------------------------------- > IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter > Gesch?ftsf?hrung: Martina Koederitz (Vorsitzende), Susanne Peter, Norbert > Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, > HRB 14562 / WEEE-Reg.-Nr. DE 99369940 > > > > From: Damir Krstic > To: gpfsug main discussion list > Date: 06/20/2016 05:10 PM > Subject: [gpfsug-discuss] ESS GL6 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > ------------------------------ > > > > Couple of questions regarding Spectrum Scale 4.2 and ESS. We recently got > our ESS delivered and are putting it in production this week. Previous to > ESS we ran GPFS 3.5 and IBM DCS3700 storage arrays. > > My question about ESS and Spectrum Scale has to do with querying available > free space and adding capacity to existing file system. > > In the old days of GPFS 3.5 I would create LUNs on 3700, zone them to > appropriate hosts, and then see them as multipath devices on NSD servers. > After that, I would create NSD disks and add them to the filesystem. > > With the ESS, however, I don't think the process is quite the same. IBM > tech that was here installing the system has created all the "LUNs" or the > equivalent in the ESS system. How do you I query what space is available to > add to the existing filesystems, and then how do you actually add space? > > I am reading ESS RedBook but the answers are not obvious. > > Thanks, > Damir _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Mon Jun 20 20:35:54 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Mon, 20 Jun 2016 19:35:54 +0000 Subject: [gpfsug-discuss] easiest way of restarting Spectrum Scale 4.2 web interface Message-ID: My ESS/Spectrum Scale web interface is spinning with "loading" screen without ever prompting me for username and password. I was able to log in early last week but something has happened since last week to make it stop working. Is there a quick way of reloading web interface to see if that fixes it before I submit help ticket with IBM? Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Mon Jun 20 20:38:03 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Mon, 20 Jun 2016 19:38:03 +0000 Subject: [gpfsug-discuss] easiest way of restarting Spectrum Scale 4.2 web interface In-Reply-To: References: Message-ID: "systemctl restart gpfsgui" -jf man. 20. jun. 2016 kl. 21.36 skrev Damir Krstic : > My ESS/Spectrum Scale web interface is spinning with "loading" screen > without ever prompting me for username and password. I was able to log in > early last week but something has happened since last week to make it stop > working. > > Is there a quick way of reloading web interface to see if that fixes it > before I submit help ticket with IBM? > > Thanks, > Damir > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gaurang.tapase at in.ibm.com Wed Jun 22 06:25:47 2016 From: gaurang.tapase at in.ibm.com (Gaurang Tapase) Date: Wed, 22 Jun 2016 10:55:47 +0530 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Message-ID: Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 22 11:33:40 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 22 Jun 2016 10:33:40 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: Message-ID: Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) -------------------------------------------------------------------------Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.kidger at uk.ibm.com Wed Jun 22 11:39:06 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 22 Jun 2016 10:39:06 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: Message-ID: Does anyone in the field have much experience with using file heat for migration, whether for object or more generically? In particular using policies to move files both ways dependant on recent usage patterns. And also if you ever move files to colder storage without necessarily waiting until the files are say over 7 days old, since you know they are not going to be used for a while. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 12:34:11, daniel.kidger at uk.ibm.com wrote: From: daniel.kidger at uk.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: Date: 22 Jun 2016 12:34:11 Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AUUnless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Wed Jun 22 12:30:10 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 22 Jun 2016 11:30:10 +0000 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Message-ID: <1F32A0BF-7B3B-4407-B0D0-1F05A4B144ED@nuance.com> Hi Daniel I've done a fair amount of work with GPFS "File Heat" and policy migration here at Nuance. Let me know if you want to discuss it a bit more. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid robert.oesterlin @ nuance.com From: on behalf of Daniel Kidger Reply-To: gpfsug main discussion list Date: Wednesday, June 22, 2016 at 5:39 AM To: gpfsug main discussion list Cc: gpfsug main discussion list Subject: [EXTERNAL] Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Does anyone in the field have much experience with using file heat for migration, whether for object or more generically? In particular using policies to move files both ways dependant on recent usage patterns. And also if you ever move files to colder storage without necessarily waiting until the files are say over 7 days old, since you know they are not going to be used for a while. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse ________________________________ On 22 Jun 2016, 12:34:11, daniel.kidger at uk.ibm.com wrote: From: daniel.kidger at uk.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: Date: 22 Jun 2016 12:34:11 Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Gaurang, Nice blog .. I must try and find some time to set a system up and try it for myself when I am not travelling ( which seems to be all the time at the moment). Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse ________________________________ On 22 Jun 2016, 07:26:21, gaurang.tapase at in.ibm.com wrote: From: gaurang.tapase at in.ibm.com To: gpfsug-discuss at spectrumscale.org Cc: sandeep.patil at in.ibm.com, smita.raut at in.ibm.com Date: 22 Jun 2016 07:26:21 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Hello, Based on inputs from some of the Spectrum Scale users, sharing a blog link for "Automatic tiering of object data based on object heat" by Smita. https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ Regards, Gaurang ------------------------------------------------------------------------ Gaurang S Tapase Spectrum Scale & OpenStack Development IBM India Storage Lab, Pune (India) Email : gaurang.tapase at in.ibm.com Phone : +91-20-42025699 (W), +91-9860082042(Cell) ------------------------------------------------------------------------- Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonathan at buzzard.me.uk Wed Jun 22 13:21:54 2016 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Wed, 22 Jun 2016 13:21:54 +0100 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: References: Message-ID: <1466598114.3773.201.camel@buzzard.phy.strath.ac.uk> On Wed, 2016-06-22 at 10:39 +0000, Daniel Kidger wrote: > Does anyone in the field have much experience with using file heat for > migration, whether for object or more generically? In particular using > policies to move files both ways dependant on recent usage patterns. > In my experience moving files from slow to fast storage (I am ignoring a tape based or other really slow layer here) was generally a waste of time for three reasons. Firstly by the time you notice that the file is in use again 99% of the time user has stopped using it anyway so there is no advantage to moving it back to faster storage. You would need to scan the file system more than once a day to capture more reused files in my experience. Secondly if the data is modified and saved due to the way most software handles this you end up with a "new" file which will land in the fast storage anyway. That is most software on a save creates a new temporary file first with the new version of the file, renames the old file to something temporary, renames the new file to the right file name and then finally deletes the old version of file. That way if the program crashes somewhere in the save there will always be a good version of the file to go back to. Thirdly the slower tier (I am thinking along the lines of large RAID6 or equivalent devices) generally has lots of spare IOPS capacity anyway. Consequently the small amount or "revisited" data that does remain in use beyond a day is not going to have a significant impact. I measured around 10 IOPS on average on the slow disk in a general purpose file system. The only time it broke this was when a flush of data form the faster tier arrived at which point they would peg out at around 120 IOPS which is what you would expect for a 7200 RPM disk. > > And also if you ever move files to colder storage without necessarily > waiting until the files are say over 7 days old, since you know they > are not going to be used for a while. > I would do things like punt .iso images directly to slower storage. Those sorts of files generally don't benefit from having low seek times which is what your fast disk pools give you. I would suggest that video files like MPEG and MP4 could be potentially also be treated similarly. So a rule like the following /* force ISO images onto nearline storage */ RULE 'iso' SET POOL 'slow' WHERE LOWER(NAME) LIKE '%.iso' What you might punt to slower storage will likely depend heavily on what you file system is being used for. You can of course also use this to "discourage" your storage from being used for "personal" use by giving certain file types lower performance. So a rule like the following puts all the music files on slower storage. /* force MP3's and the like onto nearline storage forever */ RULE 'mp3' SET POOL 'slow' WHERE LOWER(NAME) LIKE '%.mp3' OR LOWER(NAME) LIKE '%.m4a' OR LOWER(NAME) LIKE '%.wma' JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From chekh at stanford.edu Fri Jun 24 19:09:59 2016 From: chekh at stanford.edu (Alex Chekholko) Date: Fri, 24 Jun 2016 11:09:59 -0700 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: References: Message-ID: <576D7777.50609@stanford.edu> On 06/21/2016 10:25 PM, Gaurang Tapase wrote: > Hello, > > Based on inputs from some of the Spectrum Scale users, sharing a blog > link for "Automatic tiering of object data based on object heat" by Smita. > https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ > Thanks for the blog post. I wanted to post my experiences: Note "sata0" is really NL-SAS. [root at scg-gs0 ~]# df -h|grep gsfs /dev/gsfs0 4.4P 3.1P 1.4P 70% /srv/gsfs0 [root at scg-gs0 ~]# df -i|grep gsfs /dev/gsfs0 314572800 234237984 80334816 75% /srv/gsfs0 [root at scg-gs0 ~]# mmlspool gsfs0 Storage pools in file system at '/srv/gsfs0': Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB) system 0 256 KB yes yes 1887436800 118287616 ( 6%) 1887436800 328248832 ( 17%) sas0 65537 256 KB yes no 259795189760 59495012864 ( 23%) 0 0 ( 0%) sata0 65538 256 KB yes no 4523875631104 1405369561600 ( 31%) 0 0 ( 0%) [root at scg-gs0 ~]# mmlspolicy gsfs0 -L RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' [root at scg-gs0 ~]# cat policy_gpool.txt rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' LIMIT(80) THEN 'sata0' rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT) [root at scg-gs0 ~]# mmlsconfig|grep -i heat fileHeatPeriodMinutes 1440 fileHeatLossPercent 1 The first thing to know is that the MIGRATE rules only run when you run 'mmapplypolicy', even is you have the migrate policy loaded. So you have to decide when/how to do that. The filesystem scan takes a while; here are the parameters that let a full scan of our filesystem complete in under 10mins (8 I/O servers, ~300M inodes): mmapplypolicy gsfs0 -P policy.txt -N scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g /srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4 The issues we have are: 1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD, 259TB 10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% ?fast?, 94.57% ?slow?) 2) the client I/O can max out the IOPS in the "slow" pool and in the "fast" pool The original idea was to have all client I/O go to the 10K pool, then migrate least-used data to slow pool. Unfortunately our fast pool fills up too quickly and then GPFS can't empty it quickly enough into the slow pool. So I end up switching the placement policy to put files into the slow pool. And then emptying the fast pool into the slow pool doesn't work too well because the slow pool is already maxed out on IOPS just handling the workload. So moving the data from fast to slow takes weeks and impacts performance. The original intention was: [root at scg-gs0 ~]# cat policy_placement.txt RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%' RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' I would recommend that your total workload fits in both capacity and performance into your "primary" aka "fast" pool. I also just learned recently of the LIMIT parameter for placement, e.g. RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' LIMIT (95) RULE 'spillover' SET POOL 'sata0' So that would help with the pool not filling, but won't help with not having enough I/O for migration+clients. Regards, -- Alex Chekholko chekh at stanford.edu From kraemerf at de.ibm.com Sun Jun 26 12:04:42 2016 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Sun, 26 Jun 2016 13:04:42 +0200 Subject: [gpfsug-discuss] FIY - Efficient data tiering with the Swift High Latency Middleware (SwiftHLM) Message-ID: https://developer.ibm.com/open/2016/06/21/efficient-data-tiering-swift-high-latency-middleware-swifthlm/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Sun Jun 26 16:22:46 2016 From: damir.krstic at gmail.com (Damir Krstic) Date: Sun, 26 Jun 2016 15:22:46 +0000 Subject: [gpfsug-discuss] verbs rdma errors in logs Message-ID: We recently enabled verbs/rdma on our IB network (previously we used IPoIB exclusively) and now are getting all sorts of errors/warning in logs: Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA rdma read error IBV_WC_RETRY_EXC_ERR to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 vendor_err 129 Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 1589 Jun 25 20:40:05 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.124.12 (qnode4054-ib0.quest) on mlx5_0 port 1 fabnum 0 index 1417 Jun 25 18:30:01 ems1 root: mmfs: [N] VERBS RDMA closed connection to 172.41.130.131 (qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 195 Jun 25 18:28:23 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.130.131 (qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 1044 Something to note (not sure if this is important or not) is that our ESS storage cluster and our login nodes are in connected mode with 64K MTU and all compute nodes are in datagram mode with 2.4K MTU. Are these messages something to be concerned about? Cluster seems to be performing well and although there are some node ejections, they do not seem higher than before we turned on verbs/rdma. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From aaron.s.knister at nasa.gov Sun Jun 26 17:14:10 2016 From: aaron.s.knister at nasa.gov (Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]) Date: Sun, 26 Jun 2016 16:14:10 +0000 Subject: [gpfsug-discuss] verbs rdma errors in logs References: [gpfsug-discuss] verbs rdma errors in logs Message-ID: <5F910253243E6A47B81A9A2EB424BBA101C368BF@NDMSMBX404.ndc.nasa.gov> Damir, We see similar things in our environment (3.5k nodes) that seem to correlate with GPFS recovery events. I did some digging at it seemed to me that these errors more or less mean the other side of the VERBS connection hung up on the other. The message format seems a little alarming but I think it's innocuous. I'm curious to hear what others have to say. -Aaron From: Damir Krstic Sent: 6/26/16, 11:23 AM To: gpfsug main discussion list Subject: [gpfsug-discuss] verbs rdma errors in logs We recently enabled verbs/rdma on our IB network (previously we used IPoIB exclusively) and now are getting all sorts of errors/warning in logs: Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA rdma read error IBV_WC_RETRY_EXC_ERR to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 vendor_err 129 Jun 25 23:41:30 gssio2 mmfs: [E] VERBS RDMA closed connection to 172.41.125.27 (qnode4111-ib0.quest) on mlx5_0 port 1 fabnum 0 due to RDMA read error IBV_WC_RETRY_EXC_ERR index 1589 Jun 25 20:40:05 gssio2 mmfs: [N] VERBS RDMA closed connection to 172.41.124.12 (qnode4054-ib0.quest) on mlx5_0 port 1 fabnum 0 index 1417 Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 195 Jun 25 qnode6131-ib0.quest.it.northwestern.edu) on mlx5_0 port 1 fabnum 0 index 1044 Something to note (not sure if this is important or not) is that our ESS storage cluster and our login nodes are in connected mode with 64K MTU and all compute nodes are in datagram mode with 2.4K MTU. Are these messages something to be concerned about? Cluster seems to be performing well and although there are some node ejections, they do not seem higher than before we turned on verbs/rdma. Thanks, Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 27 15:49:27 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 27 Jun 2016 10:49:27 -0400 Subject: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" In-Reply-To: <576D7777.50609@stanford.edu> References: <576D7777.50609@stanford.edu> Message-ID: I'd like to see comments and/or questions and/or suggestions from any others who have implemented, deployed or seriously planned using FILE_HEAT and/or GROUP POOL. We implemented this a few years ago. Since we haven't had much feedback, it's either perfect or ... ? ;-) Alex, my reading of your story is that your hardware does not match your requirements, capacity, speeds, feeds. One new possibly helpful improvement is that SS release 4.2 includes QOSio which allows one to control the IOPs consumed by migration. So you can control the performance "hit" of running mmapplypolicy. You can even control it on-the-fly without killing a long running job -- for example you can throttle it back to 100iops during prime-time day shift operations -- marc of GPFS. From: Alex Chekholko To: gpfsug-discuss at spectrumscale.org Date: 06/24/2016 02:10 PM Subject: Re: [gpfsug-discuss] Blog on "Automatic tiering of object data based on object heat" Sent by: gpfsug-discuss-bounces at spectrumscale.org On 06/21/2016 10:25 PM, Gaurang Tapase wrote: > Hello, > > Based on inputs from some of the Spectrum Scale users, sharing a blog > link for "Automatic tiering of object data based on object heat" by Smita. > https://smitaraut.wordpress.com/2016/06/16/hot-cakes-or-hot-objects-they-better-be-served-fast/ > Thanks for the blog post. I wanted to post my experiences: Note "sata0" is really NL-SAS. [root at scg-gs0 ~]# df -h|grep gsfs /dev/gsfs0 4.4P 3.1P 1.4P 70% /srv/gsfs0 [root at scg-gs0 ~]# df -i|grep gsfs /dev/gsfs0 314572800 234237984 80334816 75% /srv/gsfs0 [root at scg-gs0 ~]# mmlspool gsfs0 Storage pools in file system at '/srv/gsfs0': Name Id BlkSize Data Meta Total Data in (KB) Free Data in (KB) Total Meta in (KB) Free Meta in (KB) system 0 256 KB yes yes 1887436800 118287616 ( 6%) 1887436800 328248832 ( 17%) sas0 65537 256 KB yes no 259795189760 59495012864 ( 23%) 0 0 ( 0%) sata0 65538 256 KB yes no 4523875631104 1405369561600 ( 31%) 0 0 ( 0%) [root at scg-gs0 ~]# mmlspolicy gsfs0 -L RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' [root at scg-gs0 ~]# cat policy_gpool.txt rule 'grpdef' GROUP POOL 'gpool' IS 'system' LIMIT(80) THEN 'sas0' LIMIT(80) THEN 'sata0' rule 'repack' MIGRATE FROM POOL 'gpool' TO POOL 'gpool' WEIGHT(FILE_HEAT) [root at scg-gs0 ~]# mmlsconfig|grep -i heat fileHeatPeriodMinutes 1440 fileHeatLossPercent 1 The first thing to know is that the MIGRATE rules only run when you run 'mmapplypolicy', even is you have the migrate policy loaded. So you have to decide when/how to do that. The filesystem scan takes a while; here are the parameters that let a full scan of our filesystem complete in under 10mins (8 I/O servers, ~300M inodes): mmapplypolicy gsfs0 -P policy.txt -N scg-gs0,scg-gs1,scg-gs2,scg-gs3,scg-gs4,scg-gs5,scg-gs6,scg-gs7 -L 2 -g /srv/gsfs0/admin_stuff/ -I yes -B 500 -A 61 -a 4 The issues we have are: 1) the pools are not sized correctly. We have 3 tiers: 1.8TB SSD, 259TB 10K SAS, 4523 TB 7.2K NL-SAS (0.03% ssd, 5.4% ?fast?, 94.57% ?slow?) 2) the client I/O can max out the IOPS in the "slow" pool and in the "fast" pool The original idea was to have all client I/O go to the 10K pool, then migrate least-used data to slow pool. Unfortunately our fast pool fills up too quickly and then GPFS can't empty it quickly enough into the slow pool. So I end up switching the placement policy to put files into the slow pool. And then emptying the fast pool into the slow pool doesn't work too well because the slow pool is already maxed out on IOPS just handling the workload. So moving the data from fast to slow takes weeks and impacts performance. The original intention was: [root at scg-gs0 ~]# cat policy_placement.txt RULE 'faster01' SET POOL 'system' WHERE FILESET_NAME LIKE 'home.%' RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' I would recommend that your total workload fits in both capacity and performance into your "primary" aka "fast" pool. I also just learned recently of the LIMIT parameter for placement, e.g. RULE 'scratch01' SET POOL 'sata0' WHERE FILESET_NAME LIKE 'scratch' RULE 'default' SET POOL 'sas0' LIMIT (95) RULE 'spillover' SET POOL 'sata0' So that would help with the pool not filling, but won't help with not having enough I/O for migration+clients. Regards, -- Alex Chekholko chekh at stanford.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Mon Jun 27 17:22:25 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Mon, 27 Jun 2016 17:22:25 +0100 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: References: <57598963.3040706@pixitmedia.com> Message-ID: <577152C1.8070802@pixitmedia.com> Hi Marc, Quite. We have had coverage of (1) and (2) in the API for some time. My overriding thought is that (QoS aside) if one executes an mmapplypolicy, for whatever reason, then a certain amount of resource is consumed. This may not be preferred. So let me rephrase to the users: in your real-world working environment, would you prefer: a) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc which is a sanity check, but not a policy run, hence no load b) mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. To be clear; you can currently achieve either launch with the API as it stands. However we'd like to know what the general concensus would prefer to be the norm in such an implementation. Regarding dry run functionality this can be achieved globally as follows: setDryRun(True) or as a more granular decorator: @dryrun ... def delete_filesets(): Jez On 20/06/16 16:03, Marc A Kaplan wrote: > *Jez,* > > Regarding your recent post. Do the mmchpolicy and mmapplypolicy > commands have sufficient functionality for your purposes? > Are you suggesting some improvements? If so, what? Please provide > examples and/or specific suggestions. > > WRT your numbered items: > > (1) `mmchpolicy fsname -I policy-rules-file test` does a complete > syntax check on policy-rules-file and some other sanity checking. > > (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I > test -L 0` is another way to validate policy-rules-file. Rules like > MIGRATE and LIST that are interpreted by mmapplypolicy will be > scrutinized. For example check that each named pool is defined either > within the file system or by an EXTERNAL POOL rules. > > (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 > [other options]` will do a dry run and show what actions would be > performed on each file. > > > --marc > > ----------- > > API calls for Policies have the ability to run in 'test' or 'run' > mode. (ref: man mmapplypolicy) > > How would you expect to use dry-run functionality on a policy? > We have our own opinion, but we'd like to hear yours. > > 1) Validate the policy and print the policy content to stdout/other. > No mmapplypolicy is performed. > 2) Validate the policy and enforce mmapplypolicy with -I test > 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol > --flag1 --flag2 --etc > 4) Other > > Best regards, > > Jez > -- > Jez Tucker > Head of Research & Development > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media www.pixitmedia.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Jun 27 18:35:33 2016 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Mon, 27 Jun 2016 13:35:33 -0400 Subject: [gpfsug-discuss] Dry-Run functionality ? In-Reply-To: <577152C1.8070802@pixitmedia.com> References: <57598963.3040706@pixitmedia.com> <577152C1.8070802@pixitmedia.com> Message-ID: What I always recommend is to make a directory of test cases and run mmapplypolicy /gpfs-path/test-cases -I test (or -I yes if you want to try operating on the test files) -L 2 (or -L 6 or something in between) -P policy-rules [other-options] Because as I think you're saying mmapplypolicy gpfs-fsname -I test ... Can consume a lot of compute and IO resource, just scanning the metadata. From: Jez Tucker To: gpfsug main discussion list Date: 06/27/2016 12:22 PM Subject: Re: [gpfsug-discuss] Dry-Run functionality ? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Marc, Quite. We have had coverage of (1) and (2) in the API for some time. My overriding thought is that (QoS aside) if one executes an mmapplypolicy, for whatever reason, then a certain amount of resource is consumed. This may not be preferred. So let me rephrase to the users: in your real-world working environment, would you prefer: a) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc which is a sanity check, but not a policy run, hence no load b) mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. To be clear; you can currently achieve either launch with the API as it stands. However we'd like to know what the general concensus would prefer to be the norm in such an implementation. Regarding dry run functionality this can be achieved globally as follows: setDryRun(True) or as a more granular decorator: @dryrun ... def delete_filesets(): Jez On 20/06/16 16:03, Marc A Kaplan wrote: Jez, Regarding your recent post. Do the mmchpolicy and mmapplypolicy commands have sufficient functionality for your purposes? Are you suggesting some improvements? If so, what? Please provide examples and/or specific suggestions. WRT your numbered items: (1) `mmchpolicy fsname -I policy-rules-file test` does a complete syntax check on policy-rules-file and some other sanity checking. (2) `mmapplypolicy /gpfs-path/empty-directory -P policy-rules-file -I test -L 0` is another way to validate policy-rules-file. Rules like MIGRATE and LIST that are interpreted by mmapplypolicy will be scrutinized. For example check that each named pool is defined either within the file system or by an EXTERNAL POOL rules. (3) `mmapplypolicy /gpfs-path -P policy-rules-file -I test -L 2 [other options]` will do a dry run and show what actions would be performed on each file. --marc ----------- API calls for Policies have the ability to run in 'test' or 'run' mode. (ref: man mmapplypolicy) How would you expect to use dry-run functionality on a policy? We have our own opinion, but we'd like to hear yours. 1) Validate the policy and print the policy content to stdout/other. No mmapplypolicy is performed. 2) Validate the policy and enforce mmapplypolicy with -I test 3) Return what would be called. E.G. mmapplypolicy -P mypol.pol --flag1 --flag2 --etc 4) Other Best regards, Jez -- Jez Tucker Head of Research & Development _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media www.pixitmedia.com This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email._______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Wed Jun 29 16:38:09 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Wed, 29 Jun 2016 17:38:09 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Message-ID: <5773EB61.2040203@ugent.be> Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth From Robert.Oesterlin at nuance.com Wed Jun 29 16:41:11 2016 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Wed, 29 Jun 2016 15:41:11 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Message-ID: Nothing "official" from IBM, but I believe it's been delayed until later in July. Not sure if IBM would comment publically on it. Bob Oesterlin Sr Storage Engineer, Nuance HPC Grid From: on behalf of Kenneth Waegeman Reply-To: gpfsug main discussion list Date: Wednesday, June 29, 2016 at 10:38 AM To: "gpfsug-discuss at spectrumscale.org" Subject: [EXTERNAL] [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=CwICAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=fudqCYcU99iTF3_mD3_zr_XvAB9BGNRjbIka32m0Kdw&s=GIVzGr_ag_wl2INE5BTuO1fDQ0QOfKdo--ljaxj6FcI&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From raot at bnl.gov Wed Jun 29 16:43:25 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 11:43:25 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy Message-ID: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. From Kevin.Buterbaugh at Vanderbilt.Edu Wed Jun 29 16:46:57 2016 From: Kevin.Buterbaugh at Vanderbilt.Edu (Buterbaugh, Kevin L) Date: Wed, 29 Jun 2016 15:46:57 +0000 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: References: Message-ID: Hi Tejas, One thing that I have had occasional success with is a lazy unmount - i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does help in my experience, but it?s one thing I try before bouncing a node. HTHAL? Kevin On Jun 29, 2016, at 10:43 AM, Tejas Rao > wrote: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raot at bnl.gov Wed Jun 29 16:58:09 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 11:58:09 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: References: Message-ID: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> "umount -l" returns with "not mounted" message. See below. [root@ ~]# umount -l /gpfs01 umount: /gpfs01: not mounted [root@ ~]# umount -l /dev/gpfs01 umount: /dev/gpfs01: not mounted [root@ ~]# [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Thanks. On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: > Hi Tejas, > > One thing that I have had occasional success with is a lazy unmount - > i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does > help in my experience, but it?s one thing I try before bouncing a > node. HTHAL? > > Kevin > >> On Jun 29, 2016, at 10:43 AM, Tejas Rao > > wrote: >> >> Hi all, >> >> I cannot get the filesystem mounted again on few nodes. >> >> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >> mount: /dev/gpfs01 already mounted or /gpfs01 busy >> mmmount: Command failed. Examine previous error messages to determine >> cause. >> >> >> >> Things I have tried and checked. >> >> >> >> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >> /dev/gpfs01 and /gpfs01 are not seen anywhere. >> >> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >> See below. >> >> 3. mmumount gpfs01 -f >> >> >> A reboot of the client would probably fix it but as the client is >> used in production, this is not easy to do. >> >> >> Are there any tricks/hacks to remount the filesystem again without >> rebooting the client. >> >> >> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >> GPFS file systems >> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >> Shutting down! >> 'shutdown' command about to kill process 11431 >> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >> mmfsenv: Module mmfslinux is still in use. >> Unmount all GPFS file systems and issue the command: >> mmfsadm cleanup >> Unloading module mmfs26 >> ERROR: Module mmfs26 is in use >> mmfsenv: Error unloading module mmfs26. >> Unloading module mmfslinux >> ERROR: Module mmfslinux is in use by mmfs26 >> mmfsenv: Error unloading module mmfslinux. >> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >> [root@~]# >> >> Thanks. >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > ? > Kevin Buterbaugh - Senior System Administrator > Vanderbilt University - Advanced Computing Center for Research and > Education > Kevin.Buterbaugh at vanderbilt.edu > - (615)875-9633 > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Jun 29 17:02:27 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Jun 2016 16:02:27 +0000 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> References: , <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> Message-ID: Process running which is either from gpfs or using it. We used to see it when we ran torque from gpfs and forgot to stop it before shutting down gpfs. Or if you have dsmrecalld (for example) running on the node. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Tejas Rao [raot at bnl.gov] Sent: 29 June 2016 16:58 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy "umount -l" returns with "not mounted" message. See below. [root@ ~]# umount -l /gpfs01 umount: /gpfs01: not mounted [root@ ~]# umount -l /dev/gpfs01 umount: /dev/gpfs01: not mounted [root@ ~]# [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Thanks. On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: Hi Tejas, One thing that I have had occasional success with is a lazy unmount - i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does help in my experience, but it?s one thing I try before bouncing a node. HTHAL? Kevin On Jun 29, 2016, at 10:43 AM, Tejas Rao <raot at bnl.gov> wrote: Hi all, I cannot get the filesystem mounted again on few nodes. [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... mount: /dev/gpfs01 already mounted or /gpfs01 busy mmmount: Command failed. Examine previous error messages to determine cause. Things I have tried and checked. 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both /dev/gpfs01 and /gpfs01 are not seen anywhere. 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. See below. 3. mmumount gpfs01 -f A reboot of the client would probably fix it but as the client is used in production, this is not easy to do. Are there any tricks/hacks to remount the filesystem again without rebooting the client. [root@~]# /usr/lpp/mmfs/bin/mmshutdown Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of GPFS file systems Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons Shutting down! 'shutdown' command about to kill process 11431 Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra mmfsenv: Module mmfslinux is still in use. Unmount all GPFS file systems and issue the command: mmfsadm cleanup Unloading module mmfs26 ERROR: Module mmfs26 is in use mmfsenv: Error unloading module mmfs26. Unloading module mmfslinux ERROR: Module mmfslinux is in use by mmfs26 mmfsenv: Error unloading module mmfslinux. Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished [root@~]# Thanks. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss ? Kevin Buterbaugh - Senior System Administrator Vanderbilt University - Advanced Computing Center for Research and Education Kevin.Buterbaugh at vanderbilt.edu - (615)875-9633 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From konstantin.arnold at unibas.ch Wed Jun 29 17:05:33 2016 From: konstantin.arnold at unibas.ch (Konstantin Arnold) Date: Wed, 29 Jun 2016 18:05:33 +0200 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> References: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> Message-ID: <5773F1CD.2000205@unibas.ch> Hi Tejas, we have observed this behavior too, typically when automounter or open file handles are involved. We tried to close the file handles (kill the process that has them open), next tried to unmount the automouter path and then mmumount filesystem again. In cases where we could not find and close the open file handles (e.g. process cannot be killed anymore) or you're stuck with unmounting the autoumounter path, we had to stop GPFS and verify the kernel module gets unloaded. In case the kernel module stays loaded, our only option was to reboot the node. Best Konstantin On 06/29/2016 05:58 PM, Tejas Rao wrote: > "umount -l" returns with "not mounted" message. See below. > > > [root@ ~]# umount -l /gpfs01 > umount: /gpfs01: not mounted > > [root@ ~]# umount -l /dev/gpfs01 > umount: /dev/gpfs01: not mounted > [root@ ~]# > > > [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 > Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... > mount: /dev/gpfs01 already mounted or /gpfs01 busy > mmmount: Command failed. Examine previous error messages to determine cause. > > Thanks. > > > On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: >> Hi Tejas, >> >> One thing that I have had occasional success with is a lazy unmount - >> i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does >> help in my experience, but it?s one thing I try before bouncing a >> node. HTHAL? >> >> Kevin >> >>> On Jun 29, 2016, at 10:43 AM, Tejas Rao >>> <raot at bnl.gov> wrote: >>> >>> Hi all, >>> >>> I cannot get the filesystem mounted again on few nodes. >>> >>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >>> mount: /dev/gpfs01 already mounted or /gpfs01 busy >>> mmmount: Command failed. Examine previous error messages to determine >>> cause. >>> >>> >>> >>> Things I have tried and checked. >>> >>> >>> >>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >>> /dev/gpfs01 and /gpfs01 are not seen anywhere. >>> >>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >>> See below. >>> >>> 3. mmumount gpfs01 -f >>> >>> >>> A reboot of the client would probably fix it but as the client is >>> used in production, this is not easy to do. >>> >>> >>> Are there any tricks/hacks to remount the filesystem again without >>> rebooting the client. >>> >>> >>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >>> GPFS file systems >>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >>> Shutting down! >>> 'shutdown' command about to kill process 11431 >>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >>> mmfsenv: Module mmfslinux is still in use. >>> Unmount all GPFS file systems and issue the command: >>> mmfsadm cleanup >>> Unloading module mmfs26 >>> ERROR: Module mmfs26 is in use >>> mmfsenv: Error unloading module mmfs26. >>> Unloading module mmfslinux >>> ERROR: Module mmfslinux is in use by mmfs26 >>> mmfsenv: Error unloading module mmfslinux. >>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >>> [root@~]# >>> >>> Thanks. >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> ? >> Kevin Buterbaugh - Senior System Administrator >> Vanderbilt University - Advanced Computing Center for Research and >> Education >> Kevin.Buterbaugh at vanderbilt.edu >> - (615)875-9633 >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -- ------------------------------------------------------------------------------------------- Konstantin Arnold | University of Basel & SIB Klingelbergstrasse 50/70 | CH-4056 Basel | Phone: +41 61 267 15 82 Email: konstantin.arnold at unibas.ch From raot at bnl.gov Wed Jun 29 17:13:10 2016 From: raot at bnl.gov (Tejas Rao) Date: Wed, 29 Jun 2016 12:13:10 -0400 Subject: [gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy In-Reply-To: <5773F1CD.2000205@unibas.ch> References: <12b2f8d9-d477-5fa0-d1f9-710b0e775ab3@bnl.gov> <5773F1CD.2000205@unibas.ch> Message-ID: Yes, open file handles were probably the issue, although I did not see them in lsof. I restarted my application and now I can mount /gpfs01/ again. Thanks, Tejas. On 6/29/2016 12:05, Konstantin Arnold wrote: > Hi Tejas, > > we have observed this behavior too, typically when automounter or open > file handles are involved. We tried to close the file handles (kill > the process that has them open), next tried to unmount the automouter > path and then mmumount filesystem again. In cases where we could not > find and close the open file handles (e.g. process cannot be killed > anymore) or you're stuck with unmounting the autoumounter path, we had > to stop GPFS and verify the kernel module gets unloaded. In case the > kernel module stays loaded, our only option was to reboot the node. > > Best > Konstantin > > > > On 06/29/2016 05:58 PM, Tejas Rao wrote: >> "umount -l" returns with "not mounted" message. See below. >> >> >> [root@ ~]# umount -l /gpfs01 >> umount: /gpfs01: not mounted >> >> [root@ ~]# umount -l /dev/gpfs01 >> umount: /dev/gpfs01: not mounted >> [root@ ~]# >> >> >> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >> Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ... >> mount: /dev/gpfs01 already mounted or /gpfs01 busy >> mmmount: Command failed. Examine previous error messages to determine >> cause. >> >> Thanks. >> >> >> On 6/29/2016 11:46, Buterbaugh, Kevin L wrote: >>> Hi Tejas, >>> >>> One thing that I have had occasional success with is a lazy unmount - >>> i.e. ?unmount -l /gpfs01?. Doesn?t help more of the time than it does >>> help in my experience, but it?s one thing I try before bouncing a >>> node. HTHAL? >>> >>> Kevin >>> >>>> On Jun 29, 2016, at 10:43 AM, Tejas Rao >>>> <raot at bnl.gov> wrote: >>>> >>>> Hi all, >>>> >>>> I cannot get the filesystem mounted again on few nodes. >>>> >>>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01 >>>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ... >>>> mount: /dev/gpfs01 already mounted or /gpfs01 busy >>>> mmmount: Command failed. Examine previous error messages to determine >>>> cause. >>>> >>>> >>>> >>>> Things I have tried and checked. >>>> >>>> >>>> >>>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both >>>> /dev/gpfs01 and /gpfs01 are not seen anywhere. >>>> >>>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules. >>>> See below. >>>> >>>> 3. mmumount gpfs01 -f >>>> >>>> >>>> A reboot of the client would probably fix it but as the client is >>>> used in production, this is not easy to do. >>>> >>>> >>>> Are there any tricks/hacks to remount the filesystem again without >>>> rebooting the client. >>>> >>>> >>>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown >>>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of >>>> GPFS file systems >>>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons >>>> Shutting down! >>>> 'shutdown' command about to kill process 11431 >>>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra >>>> mmfsenv: Module mmfslinux is still in use. >>>> Unmount all GPFS file systems and issue the command: >>>> mmfsadm cleanup >>>> Unloading module mmfs26 >>>> ERROR: Module mmfs26 is in use >>>> mmfsenv: Error unloading module mmfs26. >>>> Unloading module mmfslinux >>>> ERROR: Module mmfslinux is in use by mmfs26 >>>> mmfsenv: Error unloading module mmfslinux. >>>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished >>>> [root@~]# >>>> >>>> Thanks. >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >>> >>> ? >>> Kevin Buterbaugh - Senior System Administrator >>> Vanderbilt University - Advanced Computing Center for Research and >>> Education >>> Kevin.Buterbaugh at vanderbilt.edu >>> - (615)875-9633 >>> >>> >>> >>> >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> > From S.J.Thompson at bham.ac.uk Wed Jun 29 17:18:50 2016 From: S.J.Thompson at bham.ac.uk (Simon Thompson (Research Computing - IT Services)) Date: Wed, 29 Jun 2016 16:18:50 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: <5773EB61.2040203@ugent.be> References: <5773EB61.2040203@ugent.be> Message-ID: I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From daniel.kidger at uk.ibm.com Wed Jun 29 17:25:07 2016 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Wed, 29 Jun 2016 16:25:07 +0000 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: Message-ID: Informally I hear that we a still a couple of weeks away. Daniel IBM Spectrum Storage Software +44 (0)7818 522266 Sent from my iPad using IBM Verse On 29 Jun 2016, 18:19:07, S.J.Thompson at bham.ac.uk wrote: From: S.J.Thompson at bham.ac.uk To: gpfsug-discuss at spectrumscale.org Cc: Date: 29 Jun 2016 18:19:07 Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discussUnless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: From stschmid at de.ibm.com Thu Jun 30 11:46:16 2016 From: stschmid at de.ibm.com (Stefan Schmidt) Date: Thu, 30 Jun 2016 12:46:16 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: References: <5773EB61.2040203@ugent.be> Message-ID: The GA date moved to July 29th. A gating issue was found late during the testing and nobody wanted to risk to release the product with this issue and provide enough time to make sure the issue is fixed and well tested. Mit freundlichen Gr??en / Kind regards Stefan Schmidt Scrum Master IBM Spectrum Scale GUI / Senior IT Architect /PMP - Dept. M069 / IBM Spectrum Scale Software Development IBM Systems Group IBM Deutschland Phone: +49-6131-84-3465 IBM Deutschland Mobile: +49-170-6346601 Hechtsheimer Str. 2 E-Mail: stschmid at de.ibm.com 55131 Mainz Germany IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Martina Koederitz Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Simon Thompson (Research Computing - IT Services)" To: gpfsug main discussion list Date: 29.06.2016 18:19 Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 Sent by: gpfsug-discuss-bounces at spectrumscale.org I'm not sure they announced... Someone may have said that they expected June, you did read all the disclaimer in the slide decks right ;) But seriously, its not a formal product announcement platform, so whilst people may talk about features, ideas etc... It shouldn't be held as a commitment. Simon ________________________________________ From: gpfsug-discuss-bounces at spectrumscale.org [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth Waegeman [kenneth.waegeman at ugent.be] Sent: 29 June 2016 16:38 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 Hi all, On the User meeting in London a month ago, they announced that SS 4.2.1 would be released somewhere in June. Since I did not found any release notes or packages at the passport advantage portal, I guess it it is not yet released? Does someone knows what would be the timing for this? Thank you! Cheers, Kenneth _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From kenneth.waegeman at ugent.be Thu Jun 30 11:56:04 2016 From: kenneth.waegeman at ugent.be (Kenneth Waegeman) Date: Thu, 30 Jun 2016 12:56:04 +0200 Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 In-Reply-To: References: <5773EB61.2040203@ugent.be> Message-ID: <5774FAC4.5060705@ugent.be> Thanks for all the info, it's good to know! Kind regards, Kenneth On 30/06/16 12:46, Stefan Schmidt wrote: > The GA date moved to July 29th. A gating issue was found late during > the testing and nobody wanted to risk to release the product with this > issue and provide enough time to make sure the issue is fixed and well > tested. > > Mit freundlichen Gr??en / Kind regards > > *Stefan Schmidt* > > Scrum Master IBM Spectrum Scale GUI / Senior IT Architect /PMP - Dept. > M069 / IBM Spectrum Scale Software Development > > IBM Systems Group > > IBM Deutschland > > ------------------------------------------------------------------------ > > Phone: +49-6131-84-3465 IBM Deutschland > Mobile: +49-170-6346601 Hechtsheimer Str. 2 > E-Mail: stschmid at de.ibm.com 55131 Mainz > > > Germany > ------------------------------------------------------------------------ > > IBM Deutschland Research & Development GmbH / Vorsitzende des > Aufsichtsrats: Martina Koederitz > Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht > Stuttgart, HRB 243294 > > > > > > > > From: "Simon Thompson (Research Computing - IT Services)" > > To: gpfsug main discussion list > Date: 29.06.2016 18:19 > Subject: Re: [gpfsug-discuss] Spectrum Scale 4.2.1 > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------ > > > > > I'm not sure they announced... Someone may have said that they > expected June, you did read all the disclaimer in the slide decks right ;) > > But seriously, its not a formal product announcement platform, so > whilst people may talk about features, ideas etc... It shouldn't be > held as a commitment. > > Simon > ________________________________________ > From: gpfsug-discuss-bounces at spectrumscale.org > [gpfsug-discuss-bounces at spectrumscale.org] on behalf of Kenneth > Waegeman [kenneth.waegeman at ugent.be] > Sent: 29 June 2016 16:38 > To: gpfsug-discuss at spectrumscale.org > Subject: [gpfsug-discuss] Spectrum Scale 4.2.1 > > Hi all, > > On the User meeting in London a month ago, they announced that SS 4.2.1 > would be released somewhere in June. Since I did not found any release > notes or packages at the passport advantage portal, I guess it it is not > yet released? Does someone knows what would be the timing for this? > > Thank you! > > Cheers, > Kenneth > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From service at metamodul.com Thu Jun 30 12:53:11 2016 From: service at metamodul.com (- -) Date: Thu, 30 Jun 2016 13:53:11 +0200 (CEST) Subject: [gpfsug-discuss] Generic HA Environment Message-ID: <743599299.124585.fca32d27-f514-4edd-af56-0af407bd9929.open-xchange@email.1und1.de> Hi IBM folks, i would like to know if the current High Available (HA) infrastructure used by the Cluster Export Services (ces) like SMB & NFS can be used for private extensions meaning i am able to add additional service by my own. Thus there would be no need to write my own HA Infrastructre. tia Hajo From Mark.Bush at siriuscom.com Thu Jun 30 15:01:46 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Jun 2016 14:01:46 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store Message-ID: I know in the trial VM it recommends using CyberDuck to get to the object side of scale. Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush | Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions | www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions -------------- next part -------------- An HTML attachment was scrubbed... URL: From janfrode at tanso.net Thu Jun 30 15:29:17 2016 From: janfrode at tanso.net (Jan-Frode Myklebust) Date: Thu, 30 Jun 2016 14:29:17 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: The object side is just openstack swift, so f.ex: https://github.com/openstack/python-swiftclient/blob/master/README.rst -jf tor. 30. jun. 2016 kl. 16.01 skrev Mark.Bush at siriuscom.com < Mark.Bush at siriuscom.com>: > I know in the trial VM it recommends using CyberDuck to get to the object > side of scale. Does anyone have a programmatic example of doing this? > Python would be preferred here. > > > > Mark R. Bush | Solutions Architect > Mobile: 210.237.8415 | mark.bush at siriuscom.com > Sirius Computer Solutions | www.siriuscom.com > 10100 Reunion Place, Suite 500, San Antonio, TX 78216 > > > > This message (including any attachments) is intended only for the use of > the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, and > exempt from disclosure under applicable law. If you are not the intended > recipient, you are hereby notified that any use, dissemination, > distribution, or copying of this communication is strictly prohibited. This > message may be viewed by parties at Sirius Computer Solutions other than > those named in the message header. This message does not contain an > official representation of Sirius Computer Solutions. If you have received > this communication in error, notify Sirius Computer Solutions immediately > and (i) destroy this message if a facsimile or (ii) delete this message > immediately if this is an electronic communication. Thank you. > Sirius Computer Solutions > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jtucker at pixitmedia.com Thu Jun 30 15:33:31 2016 From: jtucker at pixitmedia.com (Jez Tucker) Date: Thu, 30 Jun 2016 15:33:31 +0100 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: <57752DBB.3070403@pixitmedia.com> Hi Depending what your endpoint object store type is... https://github.com/openstack/python-swiftclient http://boto.cloudhackers.com/en/latest/ Though, I highly recommend: https://libcloud.apache.org/ I.E. http://libcloud.readthedocs.io/en/latest/storage/examples.html Best, Jez On 30/06/16 15:01, Mark.Bush at siriuscom.com wrote: > > I know in the trial VM it recommends using CyberDuck to get to the > object side of scale. Does anyone have a programmatic example of > doing this? Python would be preferred here. > > Mark R. Bush | Solutions Architect > Mobile: 210.237.8415 | mark.bush at siriuscom.com > Sirius Computer Solutions | www.siriuscom.com > 10100 Reunion Place, Suite 500, San Antonio, TX 78216 > > This message (including any attachments) is intended only for the use > of the individual or entity to which it is addressed and may contain > information that is non-public, proprietary, privileged, confidential, > and exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any use, > dissemination, distribution, or copying of this communication is > strictly prohibited. This message may be viewed by parties at Sirius > Computer Solutions other than those named in the message header. This > message does not contain an official representation of Sirius Computer > Solutions. If you have received this communication in error, notify > Sirius Computer Solutions immediately and (i) destroy this message if > a facsimile or (ii) delete this message immediately if this is an > electronic communication. Thank you. > > Sirius Computer Solutions > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media / ArcaStream www.pixitmedia.com www.arcastream.com -- This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Mark.Bush at siriuscom.com Thu Jun 30 15:48:00 2016 From: Mark.Bush at siriuscom.com (Mark.Bush at siriuscom.com) Date: Thu, 30 Jun 2016 14:48:00 +0000 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: <57752DBB.3070403@pixitmedia.com> References: <57752DBB.3070403@pixitmedia.com> Message-ID: <19124267-08D0-4232-BBF0-835653D8D4BB@siriuscom.com> Thanks to Jan-Frode and Jez. I will use the import swiftclient mechanism for this. From: Jez Tucker Organization: Pixit Media Limited Reply-To: gpfsug main discussion list Date: Thursday, June 30, 2016 at 9:33 AM To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] restful way to get at SS OBJ store Hi Depending what your endpoint object store type is... https://github.com/openstack/python-swiftclient http://boto.cloudhackers.com/en/latest/ Though, I highly recommend: https://libcloud.apache.org/ I.E. http://libcloud.readthedocs.io/en/latest/storage/examples.html Best, Jez On 30/06/16 15:01, Mark.Bush at siriuscom.com wrote: I know in the trial VM it recommends using CyberDuck to get to the object side of scale. Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush | Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions | www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -- Jez Tucker Head of Research & Development Pixit Media / ArcaStream www.pixitmedia.com www.arcastream.com [http://pixitmedia.com/sig/sig-cio.jpg] This email is confidential in that it is intended for the exclusive attention of the addressee(s) indicated. If you are not the intended recipient, this email should not be read or disclosed to any other person. Please notify the sender immediately and delete this email from your computer system. Any opinions expressed are not necessarily those of the company from which this email was sent and, whilst to the best of our knowledge no viruses or defects exist, no responsibility can be accepted for any loss or damage arising from its receipt or subsequent use of this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From billowen at us.ibm.com Thu Jun 30 16:37:05 2016 From: billowen at us.ibm.com (Bill Owen) Date: Thu, 30 Jun 2016 08:37:05 -0700 Subject: [gpfsug-discuss] restful way to get at SS OBJ store In-Reply-To: References: Message-ID: Also see: http://docs.openstack.org/developer/swift/api/object_api_v1_overview.html The trial VM has python-swiftclient installed. You can execute basic swift commands as follows: source openrc #define environment variables for account, user credentials etc. swift stat #show account statistics swift upload One helpful trick is to execute the client commands with --debug flag. This will show the exact http request that the client code is sending. For example: # swift --debug stat INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): client28 DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_7c0243a523cf436e9a0e79c9b7083f9e HTTP/1.1" 204 0 DEBUG:swiftclient:REQ: curl -i http://client28:8080/v1/AUTH_7c0243a523cf436e9a0e79c9b7083f9e -I -H "X-Auth-Token: " DEBUG:swiftclient:RESP STATUS: 204 No Content DEBUG:swiftclient:RESP HEADERS: [('content-length', '0'), ('x-account-object-count', '0'), ('x-account-project-domain-id', 'default'), ('x-account-storage-policy-policy-0-bytes-used', '0'), ('x-account-storage-policy-policy-0-container-count', '10'), ('x-timestamp', '1467238461.63001'), ('x-account-storage-policy-policy-0-object-count', '0'), ('x-trans-id', 'txd3e3d9ca87fe4e23a5c9c-0057753b57'), ('date', 'Thu, 30 Jun 2016 15:31:36 GMT'), ('x-account-bytes-used', '0'), ('x-account-container-count', '10'), ('content-type', 'text/plain; charset=utf-8'), ('accept-ranges', 'bytes')] Account: AUTH_7c0243a523cf436e9a0e79c9b7083f9e Containers: 10 Objects: 0 Bytes: 0 Containers in policy "policy-0": 10 Objects in policy "policy-0": 0 Bytes in policy "policy-0": 0 X-Account-Project-Domain-Id: default X-Timestamp: 1467238461.63001 X-Trans-Id: txd3e3d9ca87fe4e23a5c9c-0057753b57 Content-Type: text/plain; charset=utf-8 Accept-Ranges: bytes Regards, Bill Owen billowen at us.ibm.com Spectrum Scale Object Storage 520-799-4829 From: Jan-Frode Myklebust To: "gpfsug-discuss at spectrumscale.org" Date: 06/30/2016 07:29 AM Subject: Re: [gpfsug-discuss] restful way to get at SS OBJ store Sent by: gpfsug-discuss-bounces at spectrumscale.org The object side is just openstack swift, so f.ex: https://github.com/openstack/python-swiftclient/blob/master/README.rst -jf tor. 30. jun. 2016 kl. 16.01 skrev Mark.Bush at siriuscom.com < Mark.Bush at siriuscom.com>: I know in the trial VM it recommends using CyberDuck to get to the object side of scale.? Does anyone have a programmatic example of doing this? Python would be preferred here. Mark R. Bush?| Solutions Architect Mobile: 210.237.8415 | mark.bush at siriuscom.com Sirius Computer Solutions?|?www.siriuscom.com 10100 Reunion Place, Suite 500, San Antonio, TX 78216 This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. This message may be viewed by parties at Sirius Computer Solutions other than those named in the message header. This message does not contain an official representation of Sirius Computer Solutions. If you have received this communication in error, notify Sirius Computer Solutions immediately and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you. Sirius Computer Solutions _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: