From stuartb at 4gh.net Wed May 15 17:12:28 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:12:28 -0400 (EDT) Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: Message-ID: On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From stuartb at 4gh.net Wed May 15 17:16:59 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:16:59 -0400 (EDT) Subject: [gpfsug-discuss] mmap I/O on GPFS Message-ID: Has anyone done mmap() I/O with GPFS? We are seeing quite variable performance issues with one of our applications (our first to use mmap I/O). We are using files larger than physical memory (32G-128G files on 24G x3650 M2 nodes). With a simple test case doing a sum of all of the bytes in a file we are not always seeing expected performance. When the problems occur the test program is spending all of its time in system wait (1 core of otherwise idle 8 core system). Increasing pagepool to 512M (from default) and/or increasing seqDiscardThreshhold to 1500G seemed to help with forward sequential reading. I'm not sure which change (or some other) is changing performance and think there is something else going on. Now our application programmer is testing by summing the bytes starting from the middle of the file and working out (alternating up and down) which may break the sequential read detection logic in GPFS. This seems to have returned to a very slow reading process. So far we are only testing with file reads but mmap() based file writes will be of interest soon. We are using GPFS 3.5.0-7 on this cluster with CentOS 6.4. The GPFS Block size is 262144. Per the FAQ/Release Notes we have set transparent_hugepage=never on our kernel boot options. I don't think this mmap problem is network related, but I'm looking at another problem which might be. The network is a mixed 10G/1G network using BNT switches. MTU is still 1500. Earlier we compared against a NetApp server and saw much better performance with the sequential read case. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From ORCUNB at tr.ibm.com Wed May 15 20:05:05 2013 From: ORCUNB at tr.ibm.com (Orcun Budak) Date: Wed, 15 May 2013 22:05:05 +0300 Subject: [gpfsug-discuss] AUTO: Orcun Budak is out of the office. (returning 20.05.2013) Message-ID: I am out of the office until 20.05.2013. Note: This is an automated response to your message "Re: [gpfsug-discuss] GPS Native RAID on linux?" sent on 15/05/2013 19:12:28. This is the only notification you will receive while this person is away. From luke.raimbach at oerc.ox.ac.uk Wed May 15 22:39:29 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Wed, 15 May 2013 21:39:29 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From mattw at vpac.org Wed May 15 23:27:41 2013 From: mattw at vpac.org (Matthew Wallis) Date: Thu, 16 May 2013 08:27:41 +1000 Subject: [gpfsug-discuss] mmap I/O on GPFS In-Reply-To: References: Message-ID: Hi Stuart, On 16/05/2013, at 2:16 AM, Stuart Barkley wrote: > Has anyone done mmap() I/O with GPFS? We've used it with GPFS under Windows, and I believe it's generally a Bad Idea?. From what I understand of mmap() I/O, you're basically opening up a file handle _on disk_ that maps to a location in memory. The problem is that those file handles are 4K in size. No striped filesystem likes 4K files. The application we were supporting was dealing with 5000 or so 110MB files. Initially GPFS performance was worse than a system with a single disk. After we convinced them to ditch mmap() I/O, our system became "the fastest we've ever run this on." So fast in fact, that we found all the other limitations on scaling that they'd built into their system. So for us mmap() IO became something of a curse word. Your mileage may vary. Matt. -- Matthew Wallis, Senior HPC Systems Administrator Victorian Partnership for Advanced Computing. Ph: +61 3 9925 4452 Fax: +61 3 9925 4647 From chris_stone at uk.ibm.com Thu May 16 10:13:43 2013 From: chris_stone at uk.ibm.com (Chris Stone) Date: Thu, 16 May 2013 10:13:43 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK :+44 (0) 1962 816199 - (Internal x246199) Mobile: +44 (0)7967 275 026 Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 758 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 549 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 111 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Thu May 16 15:32:57 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Thu, 16 May 2013 14:32:57 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: Ah such a shame. Does anyone have any idea when this might become mainstream for Linux? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Chris Stone [chris_stone at uk.ibm.com] Sent: 16 May 2013 10:13 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK [cid:_2_0A691F880A691C3C0032BBD580257B6D]:+44 (0) 1962 816199 - (Internal x246199) [cid:_2_0A6921F00A691C3C0032BBD580257B6D]Mobile: +44 (0)7967 275 026 [cid:_1_0A6924480A691C3C0032BBD580257B6D] Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org ________________________________ I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.jpg Type: image/jpeg Size: 758 bytes Desc: ATT00001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00002.jpg Type: image/jpeg Size: 549 bytes Desc: ATT00002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00003.gif Type: image/gif Size: 111 bytes Desc: ATT00003.gif URL: From jonathan at buzzard.me.uk Thu May 16 15:41:09 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 May 2013 15:41:09 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: <1368715269.20516.1.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-05-16 at 14:32 +0000, Luke Raimbach wrote: > Ah such a shame. Does anyone have any idea when this might become > mainstream for Linux? > I believe the answer from IBM is never. The GPFS native RAID needs to talk to the disk enclosures and as a consequence random disk enclosures will not be supported. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From stuartb at 4gh.net Wed May 15 17:12:28 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:12:28 -0400 (EDT) Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: Message-ID: On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From stuartb at 4gh.net Wed May 15 17:16:59 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:16:59 -0400 (EDT) Subject: [gpfsug-discuss] mmap I/O on GPFS Message-ID: Has anyone done mmap() I/O with GPFS? We are seeing quite variable performance issues with one of our applications (our first to use mmap I/O). We are using files larger than physical memory (32G-128G files on 24G x3650 M2 nodes). With a simple test case doing a sum of all of the bytes in a file we are not always seeing expected performance. When the problems occur the test program is spending all of its time in system wait (1 core of otherwise idle 8 core system). Increasing pagepool to 512M (from default) and/or increasing seqDiscardThreshhold to 1500G seemed to help with forward sequential reading. I'm not sure which change (or some other) is changing performance and think there is something else going on. Now our application programmer is testing by summing the bytes starting from the middle of the file and working out (alternating up and down) which may break the sequential read detection logic in GPFS. This seems to have returned to a very slow reading process. So far we are only testing with file reads but mmap() based file writes will be of interest soon. We are using GPFS 3.5.0-7 on this cluster with CentOS 6.4. The GPFS Block size is 262144. Per the FAQ/Release Notes we have set transparent_hugepage=never on our kernel boot options. I don't think this mmap problem is network related, but I'm looking at another problem which might be. The network is a mixed 10G/1G network using BNT switches. MTU is still 1500. Earlier we compared against a NetApp server and saw much better performance with the sequential read case. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From ORCUNB at tr.ibm.com Wed May 15 20:05:05 2013 From: ORCUNB at tr.ibm.com (Orcun Budak) Date: Wed, 15 May 2013 22:05:05 +0300 Subject: [gpfsug-discuss] AUTO: Orcun Budak is out of the office. (returning 20.05.2013) Message-ID: I am out of the office until 20.05.2013. Note: This is an automated response to your message "Re: [gpfsug-discuss] GPS Native RAID on linux?" sent on 15/05/2013 19:12:28. This is the only notification you will receive while this person is away. From luke.raimbach at oerc.ox.ac.uk Wed May 15 22:39:29 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Wed, 15 May 2013 21:39:29 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From mattw at vpac.org Wed May 15 23:27:41 2013 From: mattw at vpac.org (Matthew Wallis) Date: Thu, 16 May 2013 08:27:41 +1000 Subject: [gpfsug-discuss] mmap I/O on GPFS In-Reply-To: References: Message-ID: Hi Stuart, On 16/05/2013, at 2:16 AM, Stuart Barkley wrote: > Has anyone done mmap() I/O with GPFS? We've used it with GPFS under Windows, and I believe it's generally a Bad Idea?. From what I understand of mmap() I/O, you're basically opening up a file handle _on disk_ that maps to a location in memory. The problem is that those file handles are 4K in size. No striped filesystem likes 4K files. The application we were supporting was dealing with 5000 or so 110MB files. Initially GPFS performance was worse than a system with a single disk. After we convinced them to ditch mmap() I/O, our system became "the fastest we've ever run this on." So fast in fact, that we found all the other limitations on scaling that they'd built into their system. So for us mmap() IO became something of a curse word. Your mileage may vary. Matt. -- Matthew Wallis, Senior HPC Systems Administrator Victorian Partnership for Advanced Computing. Ph: +61 3 9925 4452 Fax: +61 3 9925 4647 From chris_stone at uk.ibm.com Thu May 16 10:13:43 2013 From: chris_stone at uk.ibm.com (Chris Stone) Date: Thu, 16 May 2013 10:13:43 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK :+44 (0) 1962 816199 - (Internal x246199) Mobile: +44 (0)7967 275 026 Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 758 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 549 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 111 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Thu May 16 15:32:57 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Thu, 16 May 2013 14:32:57 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: Ah such a shame. Does anyone have any idea when this might become mainstream for Linux? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Chris Stone [chris_stone at uk.ibm.com] Sent: 16 May 2013 10:13 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK [cid:_2_0A691F880A691C3C0032BBD580257B6D]:+44 (0) 1962 816199 - (Internal x246199) [cid:_2_0A6921F00A691C3C0032BBD580257B6D]Mobile: +44 (0)7967 275 026 [cid:_1_0A6924480A691C3C0032BBD580257B6D] Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org ________________________________ I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.jpg Type: image/jpeg Size: 758 bytes Desc: ATT00001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00002.jpg Type: image/jpeg Size: 549 bytes Desc: ATT00002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00003.gif Type: image/gif Size: 111 bytes Desc: ATT00003.gif URL: From jonathan at buzzard.me.uk Thu May 16 15:41:09 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 May 2013 15:41:09 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: <1368715269.20516.1.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-05-16 at 14:32 +0000, Luke Raimbach wrote: > Ah such a shame. Does anyone have any idea when this might become > mainstream for Linux? > I believe the answer from IBM is never. The GPFS native RAID needs to talk to the disk enclosures and as a consequence random disk enclosures will not be supported. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From stuartb at 4gh.net Wed May 15 17:12:28 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:12:28 -0400 (EDT) Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: Message-ID: On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From stuartb at 4gh.net Wed May 15 17:16:59 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:16:59 -0400 (EDT) Subject: [gpfsug-discuss] mmap I/O on GPFS Message-ID: Has anyone done mmap() I/O with GPFS? We are seeing quite variable performance issues with one of our applications (our first to use mmap I/O). We are using files larger than physical memory (32G-128G files on 24G x3650 M2 nodes). With a simple test case doing a sum of all of the bytes in a file we are not always seeing expected performance. When the problems occur the test program is spending all of its time in system wait (1 core of otherwise idle 8 core system). Increasing pagepool to 512M (from default) and/or increasing seqDiscardThreshhold to 1500G seemed to help with forward sequential reading. I'm not sure which change (or some other) is changing performance and think there is something else going on. Now our application programmer is testing by summing the bytes starting from the middle of the file and working out (alternating up and down) which may break the sequential read detection logic in GPFS. This seems to have returned to a very slow reading process. So far we are only testing with file reads but mmap() based file writes will be of interest soon. We are using GPFS 3.5.0-7 on this cluster with CentOS 6.4. The GPFS Block size is 262144. Per the FAQ/Release Notes we have set transparent_hugepage=never on our kernel boot options. I don't think this mmap problem is network related, but I'm looking at another problem which might be. The network is a mixed 10G/1G network using BNT switches. MTU is still 1500. Earlier we compared against a NetApp server and saw much better performance with the sequential read case. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From ORCUNB at tr.ibm.com Wed May 15 20:05:05 2013 From: ORCUNB at tr.ibm.com (Orcun Budak) Date: Wed, 15 May 2013 22:05:05 +0300 Subject: [gpfsug-discuss] AUTO: Orcun Budak is out of the office. (returning 20.05.2013) Message-ID: I am out of the office until 20.05.2013. Note: This is an automated response to your message "Re: [gpfsug-discuss] GPS Native RAID on linux?" sent on 15/05/2013 19:12:28. This is the only notification you will receive while this person is away. From luke.raimbach at oerc.ox.ac.uk Wed May 15 22:39:29 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Wed, 15 May 2013 21:39:29 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From mattw at vpac.org Wed May 15 23:27:41 2013 From: mattw at vpac.org (Matthew Wallis) Date: Thu, 16 May 2013 08:27:41 +1000 Subject: [gpfsug-discuss] mmap I/O on GPFS In-Reply-To: References: Message-ID: Hi Stuart, On 16/05/2013, at 2:16 AM, Stuart Barkley wrote: > Has anyone done mmap() I/O with GPFS? We've used it with GPFS under Windows, and I believe it's generally a Bad Idea?. From what I understand of mmap() I/O, you're basically opening up a file handle _on disk_ that maps to a location in memory. The problem is that those file handles are 4K in size. No striped filesystem likes 4K files. The application we were supporting was dealing with 5000 or so 110MB files. Initially GPFS performance was worse than a system with a single disk. After we convinced them to ditch mmap() I/O, our system became "the fastest we've ever run this on." So fast in fact, that we found all the other limitations on scaling that they'd built into their system. So for us mmap() IO became something of a curse word. Your mileage may vary. Matt. -- Matthew Wallis, Senior HPC Systems Administrator Victorian Partnership for Advanced Computing. Ph: +61 3 9925 4452 Fax: +61 3 9925 4647 From chris_stone at uk.ibm.com Thu May 16 10:13:43 2013 From: chris_stone at uk.ibm.com (Chris Stone) Date: Thu, 16 May 2013 10:13:43 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK :+44 (0) 1962 816199 - (Internal x246199) Mobile: +44 (0)7967 275 026 Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 758 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 549 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 111 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Thu May 16 15:32:57 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Thu, 16 May 2013 14:32:57 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: Ah such a shame. Does anyone have any idea when this might become mainstream for Linux? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Chris Stone [chris_stone at uk.ibm.com] Sent: 16 May 2013 10:13 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK [cid:_2_0A691F880A691C3C0032BBD580257B6D]:+44 (0) 1962 816199 - (Internal x246199) [cid:_2_0A6921F00A691C3C0032BBD580257B6D]Mobile: +44 (0)7967 275 026 [cid:_1_0A6924480A691C3C0032BBD580257B6D] Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org ________________________________ I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.jpg Type: image/jpeg Size: 758 bytes Desc: ATT00001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00002.jpg Type: image/jpeg Size: 549 bytes Desc: ATT00002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00003.gif Type: image/gif Size: 111 bytes Desc: ATT00003.gif URL: From jonathan at buzzard.me.uk Thu May 16 15:41:09 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 May 2013 15:41:09 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: <1368715269.20516.1.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-05-16 at 14:32 +0000, Luke Raimbach wrote: > Ah such a shame. Does anyone have any idea when this might become > mainstream for Linux? > I believe the answer from IBM is never. The GPFS native RAID needs to talk to the disk enclosures and as a consequence random disk enclosures will not be supported. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From stuartb at 4gh.net Wed May 15 17:12:28 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:12:28 -0400 (EDT) Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: Message-ID: On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From stuartb at 4gh.net Wed May 15 17:16:59 2013 From: stuartb at 4gh.net (Stuart Barkley) Date: Wed, 15 May 2013 12:16:59 -0400 (EDT) Subject: [gpfsug-discuss] mmap I/O on GPFS Message-ID: Has anyone done mmap() I/O with GPFS? We are seeing quite variable performance issues with one of our applications (our first to use mmap I/O). We are using files larger than physical memory (32G-128G files on 24G x3650 M2 nodes). With a simple test case doing a sum of all of the bytes in a file we are not always seeing expected performance. When the problems occur the test program is spending all of its time in system wait (1 core of otherwise idle 8 core system). Increasing pagepool to 512M (from default) and/or increasing seqDiscardThreshhold to 1500G seemed to help with forward sequential reading. I'm not sure which change (or some other) is changing performance and think there is something else going on. Now our application programmer is testing by summing the bytes starting from the middle of the file and working out (alternating up and down) which may break the sequential read detection logic in GPFS. This seems to have returned to a very slow reading process. So far we are only testing with file reads but mmap() based file writes will be of interest soon. We are using GPFS 3.5.0-7 on this cluster with CentOS 6.4. The GPFS Block size is 262144. Per the FAQ/Release Notes we have set transparent_hugepage=never on our kernel boot options. I don't think this mmap problem is network related, but I'm looking at another problem which might be. The network is a mixed 10G/1G network using BNT switches. MTU is still 1500. Earlier we compared against a NetApp server and saw much better performance with the sequential read case. Thanks, Stuart Barkley -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone From ORCUNB at tr.ibm.com Wed May 15 20:05:05 2013 From: ORCUNB at tr.ibm.com (Orcun Budak) Date: Wed, 15 May 2013 22:05:05 +0300 Subject: [gpfsug-discuss] AUTO: Orcun Budak is out of the office. (returning 20.05.2013) Message-ID: I am out of the office until 20.05.2013. Note: This is an automated response to your message "Re: [gpfsug-discuss] GPS Native RAID on linux?" sent on 15/05/2013 19:12:28. This is the only notification you will receive while this person is away. From luke.raimbach at oerc.ox.ac.uk Wed May 15 22:39:29 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Wed, 15 May 2013 21:39:29 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From mattw at vpac.org Wed May 15 23:27:41 2013 From: mattw at vpac.org (Matthew Wallis) Date: Thu, 16 May 2013 08:27:41 +1000 Subject: [gpfsug-discuss] mmap I/O on GPFS In-Reply-To: References: Message-ID: Hi Stuart, On 16/05/2013, at 2:16 AM, Stuart Barkley wrote: > Has anyone done mmap() I/O with GPFS? We've used it with GPFS under Windows, and I believe it's generally a Bad Idea?. From what I understand of mmap() I/O, you're basically opening up a file handle _on disk_ that maps to a location in memory. The problem is that those file handles are 4K in size. No striped filesystem likes 4K files. The application we were supporting was dealing with 5000 or so 110MB files. Initially GPFS performance was worse than a system with a single disk. After we convinced them to ditch mmap() I/O, our system became "the fastest we've ever run this on." So fast in fact, that we found all the other limitations on scaling that they'd built into their system. So for us mmap() IO became something of a curse word. Your mileage may vary. Matt. -- Matthew Wallis, Senior HPC Systems Administrator Victorian Partnership for Advanced Computing. Ph: +61 3 9925 4452 Fax: +61 3 9925 4647 From chris_stone at uk.ibm.com Thu May 16 10:13:43 2013 From: chris_stone at uk.ibm.com (Chris Stone) Date: Thu, 16 May 2013 10:13:43 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , Message-ID: Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK :+44 (0) 1962 816199 - (Internal x246199) Mobile: +44 (0)7967 275 026 Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 758 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 549 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 111 bytes Desc: not available URL: From luke.raimbach at oerc.ox.ac.uk Thu May 16 15:32:57 2013 From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach) Date: Thu, 16 May 2013 14:32:57 +0000 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: Ah such a shame. Does anyone have any idea when this might become mainstream for Linux? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Chris Stone [chris_stone at uk.ibm.com] Sent: 16 May 2013 10:13 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Luke My understanding is that GPFS Native RAID is only supported on two types of Storage. Power 775 Disk Enclosure - 384 disks -water cooled. GPFS Storage Server. You are right that the clues to this are not obvious. Here are two of them that I found in the docs. Here is an extract from the latest GPFS FAQ. Q2.18: On what platforms is the GPFS Native RAID (GNR) function supported? A2.18: On Power, GPFS for AIX V3.4.0-9 or later, and GPFS for Linux on Power V3.5.0-2 or later, support the GPFS Native RAID function on P775 servers with P775 Disk Enclosures. See https://www.ibm.com/developerworks/wikis/display/hpccentral/IBM+HPC+Clustering+with+Power+775+-+Service+Packs On System x, GNR is supported on the IBM System x GPFS Storage Server. See http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS112-218&appname=USN Here is the first line from the Chapter on Native RAID from the GPFS Advanced Admin Guide: Note that GPFS Native RAID is not supported on GPFS 3.5 without APAR IV16031. Cheers Chris Stone Senior IT Specialist, High Performance Computing Services Team IBM Systems & Technology Group Hursley Park, MP137, Winchester, Hants. SO21 2JN, UK [cid:_2_0A691F880A691C3C0032BBD580257B6D]:+44 (0) 1962 816199 - (Internal x246199) [cid:_2_0A6921F00A691C3C0032BBD580257B6D]Mobile: +44 (0)7967 275 026 [cid:_1_0A6924480A691C3C0032BBD580257B6D] Email: chris_stone at uk.ibm.com From: Luke Raimbach To: gpfsug main discussion list , Date: 15/05/2013 22:39 Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? Sent by: gpfsug-discuss-bounces at gpfsug.org ________________________________ I've got some x3650s... I want to do GNR. Is this a "special" feature that only IBM can unlock for x86? I want it for my x86. Now. Someone tell me which bit of the manual I've missed? -- Luke Raimbach IT Manager Oxford e-Research Centre 7 Keble Road, Oxford, OX1 3QG +44(0)1865 610639 ________________________________________ From: gpfsug-discuss-bounces at gpfsug.org [gpfsug-discuss-bounces at gpfsug.org] on behalf of Stuart Barkley [stuartb at 4gh.net] Sent: 15 May 2013 17:12 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPS Native RAID on linux? On Fri, 26 Apr 2013 at 11:06 -0000, Pete Smith wrote: > I thought from the presentation that this was available on linux ... but > documentation seems to indicate IBM GSS only? We have a new GSS (not yet installed, waiting on power). The GPFS nodes are what appear to be fairly ordinary x3650 M4 systems. Despite having Redhat licenses, I expect us to use our standard CentOS 6.4 on these systems (using the same update mirror and baseline configuration as our other systems). Or perhaps the question meant purchasing GPFS/GNR separate from a GSS purchase. I have no idea about that. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00001.jpg Type: image/jpeg Size: 758 bytes Desc: ATT00001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00002.jpg Type: image/jpeg Size: 549 bytes Desc: ATT00002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ATT00003.gif Type: image/gif Size: 111 bytes Desc: ATT00003.gif URL: From jonathan at buzzard.me.uk Thu May 16 15:41:09 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 16 May 2013 15:41:09 +0100 Subject: [gpfsug-discuss] GPS Native RAID on linux? In-Reply-To: References: , , Message-ID: <1368715269.20516.1.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-05-16 at 14:32 +0000, Luke Raimbach wrote: > Ah such a shame. Does anyone have any idea when this might become > mainstream for Linux? > I believe the answer from IBM is never. The GPFS native RAID needs to talk to the disk enclosures and as a consequence random disk enclosures will not be supported. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom.