From richard.lefebvre at calculquebec.ca Wed Nov 6 15:22:13 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Wed, 06 Nov 2013 10:22:13 -0500 Subject: [gpfsug-discuss] Moving/copying files from one file system to another In-Reply-To: <5271485E.80405@stanford.edu> References: <526E833B.6040402@calculquebec.ca> <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk> <5271485E.80405@stanford.edu> Message-ID: <527A5EA5.8070906@calculquebec.ca> Thank you to all that answer, I will stay with scatter. So, now I will just go with doing a series of mmadddisk and mmdeldisk in sequence. I'm not doing all the adddisk first since I down want the users to think that there disk space has doubled, it is a switch with a bit of extra. I intent to do a rebalance with the mmdeldisk commands. Since the file system is full at 97% I have a felling that if I don't rebalance, at the mmdeldisk, something will choke. Richard On 10/30/2013 01:56 PM, Alex Chekholko wrote: > On 10/30/13, 5:47 AM, Jonathan Buzzard wrote: >> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote: >> >> [SNIP] >> >>> Also, another question, under what condition a scatter allocation better >>> then cluster allocation. We currently have a cluster of 650 nodes all >>> accessing the same 230TB gpfs file system. >>> >> >> Scatter allocation is better in almost all circumstances. Basically by >> scattering the files to all corners you don't get hotspots where just a >> small subset of the disks are being hammered by lots of accesses to a >> handful of files, while the rest of the disks sit idle. >> > > If you do benchmarks with only a few threads, you will see higher > performance with 'cluster' allocation. So if your workload is only a > few clients accessing the FS in a mostly streaming way, you'd see better > performance from 'cluster'. > > With 650 nodes, even if each client is doing streaming reads, at the > filesystem level that would all be interleaved and thus be random reads. > But it's tough to do a big enough benchmark to show the difference in > performance. > > I had a tough time convincing people to use 'scatter' instead of > 'cluster' even though I think the documentation is clear about the > difference, and even gives you the sizing parameters ( greater than 8 > disks or 8 NSDs? use 'scatter'). > > We use 'scatter' now. > > Regards From crobson at ocf.co.uk Tue Nov 12 11:18:39 2013 From: crobson at ocf.co.uk (Claire Robson) Date: Tue, 12 Nov 2013 11:18:39 +0000 Subject: [gpfsug-discuss] Informal Meeting at MEW Message-ID: Dear Members, We will be having an informal group meet up at the upcoming Machine Evaluation Workshop on 28th November where we can discuss any updates from Supercomputing as well as give you an opportunity to network with other GPFS users. If you would like to attend please email me at secretary at gpfsug.org To attend the session you also need to be registered for the MEW event https://eventbooking.stfc.ac.uk/news-events/mew24 Hope to see you there! Claire Robson GPFS User Group Secretary OCF plc Tel: 0114 257 2200 Mob: 07508 033896 Fax: 0114 257 0022 Web: www.ocf.co.uk Blog: http://blog.ocf.co.uk Twitter: @ocfplc OCF plc is a company registered in England and Wales. Registered number 4132533, VAT number GB 780 6803 14. Registered office address: OCF plc, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield, S35 2PG This message is private and confidential. If you have received this message in error, please notify us immediately and remove it from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Nov 15 18:53:27 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 15 Nov 2013 13:53:27 -0500 Subject: [gpfsug-discuss] mmdeldisk is stuck Message-ID: <52866DA7.5050009@calculquebec.ca> Hi, I'm doing has I the mmdeldisk as I mention in a previous thread. But now it seems to be sort of stuck. The progress (in %) on the command line is stopped, not just showing the same %, but just stopped. mmdf shows that the amount of data free is not moving. I see with the waiters (mmlsnode -N waiters -L) a lot of of "Waiting to find big enough file for helper" on one client node, and an another group waiting on that node. Do I just have to be patient? How do I find what is really happening? Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) From secretary at gpfsug.org Mon Nov 25 11:46:27 2013 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Mon, 25 Nov 2013 11:46:27 +0000 Subject: [gpfsug-discuss] Group Updates Message-ID: Hi Members, A few updates for you: There is an informal meeting at MEW ( https://eventbooking.stfc.ac.uk/news-events/mew24) this this week in Liverpool and possibly a similar type of get together the day before the HPC SIG (http://www.hpc-sig.org/) in Edinburgh which is scheduled for 18th December. The next main/formal meeting is being planned for April 2014. Please let Jez (chair at gpfsug.org) or I know if there are any specific agenda items you would like to see and we will do our best to incorporate. Last week we re-launched the GPFS UG website (http://www.gpfsug.org/) ? do let me know what you think, all feedback is welcome as we want to include information that you, the members, want to see. I?ll be sending out more communications to the mailing list and adding updates to the website so please stay tuned! Thanks, Claire GPFS User Group Secretary -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Wed Nov 6 15:22:13 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Wed, 06 Nov 2013 10:22:13 -0500 Subject: [gpfsug-discuss] Moving/copying files from one file system to another In-Reply-To: <5271485E.80405@stanford.edu> References: <526E833B.6040402@calculquebec.ca> <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk> <5271485E.80405@stanford.edu> Message-ID: <527A5EA5.8070906@calculquebec.ca> Thank you to all that answer, I will stay with scatter. So, now I will just go with doing a series of mmadddisk and mmdeldisk in sequence. I'm not doing all the adddisk first since I down want the users to think that there disk space has doubled, it is a switch with a bit of extra. I intent to do a rebalance with the mmdeldisk commands. Since the file system is full at 97% I have a felling that if I don't rebalance, at the mmdeldisk, something will choke. Richard On 10/30/2013 01:56 PM, Alex Chekholko wrote: > On 10/30/13, 5:47 AM, Jonathan Buzzard wrote: >> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote: >> >> [SNIP] >> >>> Also, another question, under what condition a scatter allocation better >>> then cluster allocation. We currently have a cluster of 650 nodes all >>> accessing the same 230TB gpfs file system. >>> >> >> Scatter allocation is better in almost all circumstances. Basically by >> scattering the files to all corners you don't get hotspots where just a >> small subset of the disks are being hammered by lots of accesses to a >> handful of files, while the rest of the disks sit idle. >> > > If you do benchmarks with only a few threads, you will see higher > performance with 'cluster' allocation. So if your workload is only a > few clients accessing the FS in a mostly streaming way, you'd see better > performance from 'cluster'. > > With 650 nodes, even if each client is doing streaming reads, at the > filesystem level that would all be interleaved and thus be random reads. > But it's tough to do a big enough benchmark to show the difference in > performance. > > I had a tough time convincing people to use 'scatter' instead of > 'cluster' even though I think the documentation is clear about the > difference, and even gives you the sizing parameters ( greater than 8 > disks or 8 NSDs? use 'scatter'). > > We use 'scatter' now. > > Regards From crobson at ocf.co.uk Tue Nov 12 11:18:39 2013 From: crobson at ocf.co.uk (Claire Robson) Date: Tue, 12 Nov 2013 11:18:39 +0000 Subject: [gpfsug-discuss] Informal Meeting at MEW Message-ID: Dear Members, We will be having an informal group meet up at the upcoming Machine Evaluation Workshop on 28th November where we can discuss any updates from Supercomputing as well as give you an opportunity to network with other GPFS users. If you would like to attend please email me at secretary at gpfsug.org To attend the session you also need to be registered for the MEW event https://eventbooking.stfc.ac.uk/news-events/mew24 Hope to see you there! Claire Robson GPFS User Group Secretary OCF plc Tel: 0114 257 2200 Mob: 07508 033896 Fax: 0114 257 0022 Web: www.ocf.co.uk Blog: http://blog.ocf.co.uk Twitter: @ocfplc OCF plc is a company registered in England and Wales. Registered number 4132533, VAT number GB 780 6803 14. Registered office address: OCF plc, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield, S35 2PG This message is private and confidential. If you have received this message in error, please notify us immediately and remove it from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Nov 15 18:53:27 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 15 Nov 2013 13:53:27 -0500 Subject: [gpfsug-discuss] mmdeldisk is stuck Message-ID: <52866DA7.5050009@calculquebec.ca> Hi, I'm doing has I the mmdeldisk as I mention in a previous thread. But now it seems to be sort of stuck. The progress (in %) on the command line is stopped, not just showing the same %, but just stopped. mmdf shows that the amount of data free is not moving. I see with the waiters (mmlsnode -N waiters -L) a lot of of "Waiting to find big enough file for helper" on one client node, and an another group waiting on that node. Do I just have to be patient? How do I find what is really happening? Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) From secretary at gpfsug.org Mon Nov 25 11:46:27 2013 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Mon, 25 Nov 2013 11:46:27 +0000 Subject: [gpfsug-discuss] Group Updates Message-ID: Hi Members, A few updates for you: There is an informal meeting at MEW ( https://eventbooking.stfc.ac.uk/news-events/mew24) this this week in Liverpool and possibly a similar type of get together the day before the HPC SIG (http://www.hpc-sig.org/) in Edinburgh which is scheduled for 18th December. The next main/formal meeting is being planned for April 2014. Please let Jez (chair at gpfsug.org) or I know if there are any specific agenda items you would like to see and we will do our best to incorporate. Last week we re-launched the GPFS UG website (http://www.gpfsug.org/) ? do let me know what you think, all feedback is welcome as we want to include information that you, the members, want to see. I?ll be sending out more communications to the mailing list and adding updates to the website so please stay tuned! Thanks, Claire GPFS User Group Secretary -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Wed Nov 6 15:22:13 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Wed, 06 Nov 2013 10:22:13 -0500 Subject: [gpfsug-discuss] Moving/copying files from one file system to another In-Reply-To: <5271485E.80405@stanford.edu> References: <526E833B.6040402@calculquebec.ca> <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk> <5271485E.80405@stanford.edu> Message-ID: <527A5EA5.8070906@calculquebec.ca> Thank you to all that answer, I will stay with scatter. So, now I will just go with doing a series of mmadddisk and mmdeldisk in sequence. I'm not doing all the adddisk first since I down want the users to think that there disk space has doubled, it is a switch with a bit of extra. I intent to do a rebalance with the mmdeldisk commands. Since the file system is full at 97% I have a felling that if I don't rebalance, at the mmdeldisk, something will choke. Richard On 10/30/2013 01:56 PM, Alex Chekholko wrote: > On 10/30/13, 5:47 AM, Jonathan Buzzard wrote: >> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote: >> >> [SNIP] >> >>> Also, another question, under what condition a scatter allocation better >>> then cluster allocation. We currently have a cluster of 650 nodes all >>> accessing the same 230TB gpfs file system. >>> >> >> Scatter allocation is better in almost all circumstances. Basically by >> scattering the files to all corners you don't get hotspots where just a >> small subset of the disks are being hammered by lots of accesses to a >> handful of files, while the rest of the disks sit idle. >> > > If you do benchmarks with only a few threads, you will see higher > performance with 'cluster' allocation. So if your workload is only a > few clients accessing the FS in a mostly streaming way, you'd see better > performance from 'cluster'. > > With 650 nodes, even if each client is doing streaming reads, at the > filesystem level that would all be interleaved and thus be random reads. > But it's tough to do a big enough benchmark to show the difference in > performance. > > I had a tough time convincing people to use 'scatter' instead of > 'cluster' even though I think the documentation is clear about the > difference, and even gives you the sizing parameters ( greater than 8 > disks or 8 NSDs? use 'scatter'). > > We use 'scatter' now. > > Regards From crobson at ocf.co.uk Tue Nov 12 11:18:39 2013 From: crobson at ocf.co.uk (Claire Robson) Date: Tue, 12 Nov 2013 11:18:39 +0000 Subject: [gpfsug-discuss] Informal Meeting at MEW Message-ID: Dear Members, We will be having an informal group meet up at the upcoming Machine Evaluation Workshop on 28th November where we can discuss any updates from Supercomputing as well as give you an opportunity to network with other GPFS users. If you would like to attend please email me at secretary at gpfsug.org To attend the session you also need to be registered for the MEW event https://eventbooking.stfc.ac.uk/news-events/mew24 Hope to see you there! Claire Robson GPFS User Group Secretary OCF plc Tel: 0114 257 2200 Mob: 07508 033896 Fax: 0114 257 0022 Web: www.ocf.co.uk Blog: http://blog.ocf.co.uk Twitter: @ocfplc OCF plc is a company registered in England and Wales. Registered number 4132533, VAT number GB 780 6803 14. Registered office address: OCF plc, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield, S35 2PG This message is private and confidential. If you have received this message in error, please notify us immediately and remove it from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Nov 15 18:53:27 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 15 Nov 2013 13:53:27 -0500 Subject: [gpfsug-discuss] mmdeldisk is stuck Message-ID: <52866DA7.5050009@calculquebec.ca> Hi, I'm doing has I the mmdeldisk as I mention in a previous thread. But now it seems to be sort of stuck. The progress (in %) on the command line is stopped, not just showing the same %, but just stopped. mmdf shows that the amount of data free is not moving. I see with the waiters (mmlsnode -N waiters -L) a lot of of "Waiting to find big enough file for helper" on one client node, and an another group waiting on that node. Do I just have to be patient? How do I find what is really happening? Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) From secretary at gpfsug.org Mon Nov 25 11:46:27 2013 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Mon, 25 Nov 2013 11:46:27 +0000 Subject: [gpfsug-discuss] Group Updates Message-ID: Hi Members, A few updates for you: There is an informal meeting at MEW ( https://eventbooking.stfc.ac.uk/news-events/mew24) this this week in Liverpool and possibly a similar type of get together the day before the HPC SIG (http://www.hpc-sig.org/) in Edinburgh which is scheduled for 18th December. The next main/formal meeting is being planned for April 2014. Please let Jez (chair at gpfsug.org) or I know if there are any specific agenda items you would like to see and we will do our best to incorporate. Last week we re-launched the GPFS UG website (http://www.gpfsug.org/) ? do let me know what you think, all feedback is welcome as we want to include information that you, the members, want to see. I?ll be sending out more communications to the mailing list and adding updates to the website so please stay tuned! Thanks, Claire GPFS User Group Secretary -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Wed Nov 6 15:22:13 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Wed, 06 Nov 2013 10:22:13 -0500 Subject: [gpfsug-discuss] Moving/copying files from one file system to another In-Reply-To: <5271485E.80405@stanford.edu> References: <526E833B.6040402@calculquebec.ca> <1383137227.2921.4.camel@buzzard.phy.strath.ac.uk> <5271485E.80405@stanford.edu> Message-ID: <527A5EA5.8070906@calculquebec.ca> Thank you to all that answer, I will stay with scatter. So, now I will just go with doing a series of mmadddisk and mmdeldisk in sequence. I'm not doing all the adddisk first since I down want the users to think that there disk space has doubled, it is a switch with a bit of extra. I intent to do a rebalance with the mmdeldisk commands. Since the file system is full at 97% I have a felling that if I don't rebalance, at the mmdeldisk, something will choke. Richard On 10/30/2013 01:56 PM, Alex Chekholko wrote: > On 10/30/13, 5:47 AM, Jonathan Buzzard wrote: >> On Mon, 2013-10-28 at 11:31 -0400, Richard Lefebvre wrote: >> >> [SNIP] >> >>> Also, another question, under what condition a scatter allocation better >>> then cluster allocation. We currently have a cluster of 650 nodes all >>> accessing the same 230TB gpfs file system. >>> >> >> Scatter allocation is better in almost all circumstances. Basically by >> scattering the files to all corners you don't get hotspots where just a >> small subset of the disks are being hammered by lots of accesses to a >> handful of files, while the rest of the disks sit idle. >> > > If you do benchmarks with only a few threads, you will see higher > performance with 'cluster' allocation. So if your workload is only a > few clients accessing the FS in a mostly streaming way, you'd see better > performance from 'cluster'. > > With 650 nodes, even if each client is doing streaming reads, at the > filesystem level that would all be interleaved and thus be random reads. > But it's tough to do a big enough benchmark to show the difference in > performance. > > I had a tough time convincing people to use 'scatter' instead of > 'cluster' even though I think the documentation is clear about the > difference, and even gives you the sizing parameters ( greater than 8 > disks or 8 NSDs? use 'scatter'). > > We use 'scatter' now. > > Regards From crobson at ocf.co.uk Tue Nov 12 11:18:39 2013 From: crobson at ocf.co.uk (Claire Robson) Date: Tue, 12 Nov 2013 11:18:39 +0000 Subject: [gpfsug-discuss] Informal Meeting at MEW Message-ID: Dear Members, We will be having an informal group meet up at the upcoming Machine Evaluation Workshop on 28th November where we can discuss any updates from Supercomputing as well as give you an opportunity to network with other GPFS users. If you would like to attend please email me at secretary at gpfsug.org To attend the session you also need to be registered for the MEW event https://eventbooking.stfc.ac.uk/news-events/mew24 Hope to see you there! Claire Robson GPFS User Group Secretary OCF plc Tel: 0114 257 2200 Mob: 07508 033896 Fax: 0114 257 0022 Web: www.ocf.co.uk Blog: http://blog.ocf.co.uk Twitter: @ocfplc OCF plc is a company registered in England and Wales. Registered number 4132533, VAT number GB 780 6803 14. Registered office address: OCF plc, 5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield, S35 2PG This message is private and confidential. If you have received this message in error, please notify us immediately and remove it from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.lefebvre at calculquebec.ca Fri Nov 15 18:53:27 2013 From: richard.lefebvre at calculquebec.ca (Richard Lefebvre) Date: Fri, 15 Nov 2013 13:53:27 -0500 Subject: [gpfsug-discuss] mmdeldisk is stuck Message-ID: <52866DA7.5050009@calculquebec.ca> Hi, I'm doing has I the mmdeldisk as I mention in a previous thread. But now it seems to be sort of stuck. The progress (in %) on the command line is stopped, not just showing the same %, but just stopped. mmdf shows that the amount of data free is not moving. I see with the waiters (mmlsnode -N waiters -L) a lot of of "Waiting to find big enough file for helper" on one client node, and an another group waiting on that node. Do I just have to be patient? How do I find what is really happening? Richard -- Richard Lefebvre, Sys-admin, CQ, (514)343-6111 x5313 "Don't Panic" Richard.Lefebvre at calculquebec.ca -- THGTTG Calcul Quebec (calculquebec.ca) ------ Calcul Canada (computecanada.ca) From secretary at gpfsug.org Mon Nov 25 11:46:27 2013 From: secretary at gpfsug.org (Secretary GPFS UG) Date: Mon, 25 Nov 2013 11:46:27 +0000 Subject: [gpfsug-discuss] Group Updates Message-ID: Hi Members, A few updates for you: There is an informal meeting at MEW ( https://eventbooking.stfc.ac.uk/news-events/mew24) this this week in Liverpool and possibly a similar type of get together the day before the HPC SIG (http://www.hpc-sig.org/) in Edinburgh which is scheduled for 18th December. The next main/formal meeting is being planned for April 2014. Please let Jez (chair at gpfsug.org) or I know if there are any specific agenda items you would like to see and we will do our best to incorporate. Last week we re-launched the GPFS UG website (http://www.gpfsug.org/) ? do let me know what you think, all feedback is welcome as we want to include information that you, the members, want to see. I?ll be sending out more communications to the mailing list and adding updates to the website so please stay tuned! Thanks, Claire GPFS User Group Secretary -------------- next part -------------- An HTML attachment was scrubbed... URL: