From orlando.richards at ed.ac.uk Mon Sep 16 16:25:26 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Mon, 16 Sep 2013 16:25:26 +0100 Subject: [gpfsug-discuss] Multipath configurations Message-ID: <523722E6.3010603@ed.ac.uk> Hi folks, We're building a new storage service and are planning on using multipathd rather than LSI's rdac to handle the multipathing. It's all working well, but I'm looking at settling on the final parameters for the multipath.conf. In particular, the values for: * rr_min_io (1?) * failback (I think "manual" or "followover"?) * no_path_retry (guessing here - fail?) * dev_loss_tmo (guessing here - 15?) * fast_io_fail_tmo (guessing here - 10?) Does anyone have a working multipath.conf for LSI based storage systems (or others, for that matter), and/or have experience and wisdom to share on the above settings (and any others I may have missed?). Any war stories about dm-multipath to share? -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From seanlee at tw.ibm.com Wed Sep 18 17:17:10 2013 From: seanlee at tw.ibm.com (Sean S Lee) Date: Thu, 19 Sep 2013 00:17:10 +0800 Subject: [gpfsug-discuss] Re: Multipath configurations In-Reply-To: References: Message-ID: Hi Orlando Did you read your product documentation? Normally this info can be found under "Working with Multipath.conf file". Some useful info can also be found here: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014889306 Regards, Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From orlando.richards at ed.ac.uk Thu Sep 19 15:53:12 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Thu, 19 Sep 2013 15:53:12 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523722E6.3010603@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> Message-ID: <523B0FD8.1090403@ed.ac.uk> On 16/09/13 16:25, Orlando Richards wrote: > Hi folks, > > We're building a new storage service and are planning on using > multipathd rather than LSI's rdac to handle the multipathing. > > It's all working well, but I'm looking at settling on the final > parameters for the multipath.conf. In particular, the values for: > > * rr_min_io (1?) > * failback (I think "manual" or "followover"?) > * no_path_retry (guessing here - fail?) > * dev_loss_tmo (guessing here - 15?) > * fast_io_fail_tmo (guessing here - 10?) > > Does anyone have a working multipath.conf for LSI based storage systems > (or others, for that matter), and/or have experience and wisdom to share > on the above settings (and any others I may have missed?). Any war > stories about dm-multipath to share? > > Hi all, Thanks for the feedback on all this. From that, and more digging and testing, we've settled on the following multipath.conf stanzas: path_grouping_policy group_by_prio prio rdac path_checker rdac path_selector "round-robin 0" hardware_handler "1 rdac" features "2 pg_init_retries 50" # All "standard" up to here # Prevent ping-ponging of controllers, but # allow for automatic failback failback followover # Massively accelerate the failure detection time # (default settings give ~30-90 seconds, this gives ~5s) fast_io_fail_tmo 5 # Keep the /dev device entries in situ for 90 seconds, # in case of rapid recovery of paths dev_loss_tmo 90 # Don't queue traffic down a failed path no_path_retry fail # balance much more aggressively across the active paths rr_min_io 1 The primary goal was to have rapid and reliable failover in a cluster environment (without ping-ponging). The defaults from multipathd gave a 30-90 second pause in I/O every time a path went away - we've managed to get it down to ~5s with the above settings. Note that we've not tried this "in production" yet, but it has held up fine under heavy benchmark load. Along the way we discovered an odd GPFS "feature" - if some nodes in the cluster use RDAC (and thus have /dev/sdXX devices) and some use multipathd (and thus use /dev/dm-XX devices), then the nodes can either fail to find attached NSD devices (in the case of the RDAC host where the NSD's were initially created on a multipath host) or can try to talk to them down the wrong device (for instance - talking to /dev/sdXX rather than /dev/dm-XX). We just set up this mixed environment to compare rdac vs dm-multipath, and don't expect to put it into production - but it's the kind of thing which could end up cropping up in a system migrating from RDAC to dm-multipath, or vice versa. It seems that on creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or "generic" (rdac), and servers using one type can't see the other. We're testing a workaround for the "dm-multipath server accessing via /dev/sdXX" case just now - create the following (executable, root-owned) script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: #!/bin/ksh # # this script ensures that we are not using the raw /dev/sd\* devices for GPFS # but use the multipath /dev/dm-\* devices instead for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) do echo $dev generic done # skip the GPFS device discovery exit 0 except change that simple "$dev generic" echo to one which says "$dev mpp" or "$dev generic" depending on whether the device was created with dm-multipath or rdac attached hosts. The reverse also likely would work to get the rdac host to pick up the dm-multipath created nsd's (echo $dev mpp, for the /dev/sdXX devices). Thankfully, we have no plans to mix the environment - but for future reference it could be important (if ever migrating existing systems from rdac to dm-multipath, for instance). -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From jonathan at buzzard.me.uk Thu Sep 19 16:38:05 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 19 Sep 2013 16:38:05 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523B0FD8.1090403@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> <523B0FD8.1090403@ed.ac.uk> Message-ID: <1379605085.26154.4.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: [SNIP] > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > That is all covered in the GPFS documentation under "don't do that" ;-) [SNIP] > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > The recommendation is roughly umount GPFS everywhere, export the file system, change everything to dm-multipath/RDAC depending on your poison, import file system and mount again. I think you can skip the export/import by doing some fiddling to switch the NSD type. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From cexton26 at gmail.com Sat Sep 21 12:31:16 2013 From: cexton26 at gmail.com (Chris Exton) Date: Sat, 21 Sep 2013 12:31:16 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 20, Issue 3 In-Reply-To: References: Message-ID: I had a similar issue when upgrading from an RDAC based solution to multipath. I found a command that wasn't that well documented to change the device type mmchconfig updateNsdType=/tmp/disks On 20 September 2013 12:00, wrote: > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at gpfsug.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at gpfsug.org > > You can reach the person managing the list at > gpfsug-discuss-owner at gpfsug.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Multipath configurations (Orlando Richards) > 2. Re: Multipath configurations (Jonathan Buzzard) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 19 Sep 2013 15:53:12 +0100 > From: Orlando Richards > To: gpfsug-discuss at gpfsug.org > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <523B0FD8.1090403 at ed.ac.uk> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 16/09/13 16:25, Orlando Richards wrote: > > Hi folks, > > > > We're building a new storage service and are planning on using > > multipathd rather than LSI's rdac to handle the multipathing. > > > > It's all working well, but I'm looking at settling on the final > > parameters for the multipath.conf. In particular, the values for: > > > > * rr_min_io (1?) > > * failback (I think "manual" or "followover"?) > > * no_path_retry (guessing here - fail?) > > * dev_loss_tmo (guessing here - 15?) > > * fast_io_fail_tmo (guessing here - 10?) > > > > Does anyone have a working multipath.conf for LSI based storage systems > > (or others, for that matter), and/or have experience and wisdom to share > > on the above settings (and any others I may have missed?). Any war > > stories about dm-multipath to share? > > > > > > Hi all, > > Thanks for the feedback on all this. From that, and more digging and > testing, we've settled on the following multipath.conf stanzas: > > path_grouping_policy group_by_prio > prio rdac > path_checker rdac > path_selector "round-robin 0" > hardware_handler "1 rdac" > features "2 pg_init_retries 50" > # All "standard" up to here > > # Prevent ping-ponging of controllers, but > # allow for automatic failback > failback followover > # Massively accelerate the failure detection time > # (default settings give ~30-90 seconds, this gives ~5s) > fast_io_fail_tmo 5 > # Keep the /dev device entries in situ for 90 seconds, > # in case of rapid recovery of paths > dev_loss_tmo 90 > # Don't queue traffic down a failed path > no_path_retry fail > # balance much more aggressively across the active paths > rr_min_io 1 > > The primary goal was to have rapid and reliable failover in a cluster > environment (without ping-ponging). The defaults from multipathd gave a > 30-90 second pause in I/O every time a path went away - we've managed to > get it down to ~5s with the above settings. > > Note that we've not tried this "in production" yet, but it has held up > fine under heavy benchmark load. > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > > We're testing a workaround for the "dm-multipath server accessing via > /dev/sdXX" case just now - create the following (executable, root-owned) > script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: > > #!/bin/ksh > # > # this script ensures that we are not using the raw /dev/sd\* devices > for GPFS > # but use the multipath /dev/dm-\* devices instead > for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) > do > echo $dev generic > done > > # skip the GPFS device discovery > exit 0 > > > except change that simple "$dev generic" echo to one which says "$dev > mpp" or "$dev generic" depending on whether the device was created with > dm-multipath or rdac attached hosts. The reverse also likely would work > to get the rdac host to pick up the dm-multipath created nsd's (echo > $dev mpp, for the /dev/sdXX devices). > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > > > > > > > > -- > -- > Dr Orlando Richards > Information Services > IT Infrastructure Division > Unix Section > Tel: 0131 650 4994 > skype: orlando.richards > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > ------------------------------ > > Message: 2 > Date: Thu, 19 Sep 2013 16:38:05 +0100 > From: Jonathan Buzzard > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <1379605085.26154.4.camel at buzzard.phy.strath.ac.uk> > Content-Type: text/plain; charset="UTF-8" > > On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: > > [SNIP] > > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > > cluster use RDAC (and thus have /dev/sdXX devices) and some use > > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > > fail to find attached NSD devices (in the case of the RDAC host where > > the NSD's were initially created on a multipath host) or can try to talk > > to them down the wrong device (for instance - talking to /dev/sdXX > > rather than /dev/dm-XX). We just set up this mixed environment to > > compare rdac vs dm-multipath, and don't expect to put it into production > > - but it's the kind of thing which could end up cropping up in a system > > migrating from RDAC to dm-multipath, or vice versa. It seems that on > > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > > "generic" (rdac), and servers using one type can't see the other. > > > > That is all covered in the GPFS documentation under "don't do that" ;-) > > [SNIP] > > > > > Thankfully, we have no plans to mix the environment - but for future > > reference it could be important (if ever migrating existing systems from > > rdac to dm-multipath, for instance). > > > > The recommendation is roughly umount GPFS everywhere, export the file > system, change everything to dm-multipath/RDAC depending on your poison, > import file system and mount again. I think you can skip the > export/import by doing some fiddling to switch the NSD type. > > > JAB. > > -- > Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk > Fife, United Kingdom. > > > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 20, Issue 3 > ********************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From theo at tepucom.nl Thu Sep 26 20:25:12 2013 From: theo at tepucom.nl (Theo Purmer (Tepucom BV)) Date: Thu, 26 Sep 2013 21:25:12 +0200 (CEST) Subject: [gpfsug-discuss] afm setup Message-ID: <10465077.604006.1380223512803.JavaMail.root@tepucom.nl> Hi Folks We are working on a gpfs poc using afm to sync (async) two remote sites as synchronous syncing is not possible But I do not understand exactly how afm works. I have read all the available documentation (i think) Is there a drawing available of the components of an afm setup or a complete description of the setup. AFM is kinda new, IBM here in the netherlands does not have a tech that knows how it works. Thanks theo -------------- next part -------------- An HTML attachment was scrubbed... URL: From orlando.richards at ed.ac.uk Mon Sep 16 16:25:26 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Mon, 16 Sep 2013 16:25:26 +0100 Subject: [gpfsug-discuss] Multipath configurations Message-ID: <523722E6.3010603@ed.ac.uk> Hi folks, We're building a new storage service and are planning on using multipathd rather than LSI's rdac to handle the multipathing. It's all working well, but I'm looking at settling on the final parameters for the multipath.conf. In particular, the values for: * rr_min_io (1?) * failback (I think "manual" or "followover"?) * no_path_retry (guessing here - fail?) * dev_loss_tmo (guessing here - 15?) * fast_io_fail_tmo (guessing here - 10?) Does anyone have a working multipath.conf for LSI based storage systems (or others, for that matter), and/or have experience and wisdom to share on the above settings (and any others I may have missed?). Any war stories about dm-multipath to share? -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From seanlee at tw.ibm.com Wed Sep 18 17:17:10 2013 From: seanlee at tw.ibm.com (Sean S Lee) Date: Thu, 19 Sep 2013 00:17:10 +0800 Subject: [gpfsug-discuss] Re: Multipath configurations In-Reply-To: References: Message-ID: Hi Orlando Did you read your product documentation? Normally this info can be found under "Working with Multipath.conf file". Some useful info can also be found here: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014889306 Regards, Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From orlando.richards at ed.ac.uk Thu Sep 19 15:53:12 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Thu, 19 Sep 2013 15:53:12 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523722E6.3010603@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> Message-ID: <523B0FD8.1090403@ed.ac.uk> On 16/09/13 16:25, Orlando Richards wrote: > Hi folks, > > We're building a new storage service and are planning on using > multipathd rather than LSI's rdac to handle the multipathing. > > It's all working well, but I'm looking at settling on the final > parameters for the multipath.conf. In particular, the values for: > > * rr_min_io (1?) > * failback (I think "manual" or "followover"?) > * no_path_retry (guessing here - fail?) > * dev_loss_tmo (guessing here - 15?) > * fast_io_fail_tmo (guessing here - 10?) > > Does anyone have a working multipath.conf for LSI based storage systems > (or others, for that matter), and/or have experience and wisdom to share > on the above settings (and any others I may have missed?). Any war > stories about dm-multipath to share? > > Hi all, Thanks for the feedback on all this. From that, and more digging and testing, we've settled on the following multipath.conf stanzas: path_grouping_policy group_by_prio prio rdac path_checker rdac path_selector "round-robin 0" hardware_handler "1 rdac" features "2 pg_init_retries 50" # All "standard" up to here # Prevent ping-ponging of controllers, but # allow for automatic failback failback followover # Massively accelerate the failure detection time # (default settings give ~30-90 seconds, this gives ~5s) fast_io_fail_tmo 5 # Keep the /dev device entries in situ for 90 seconds, # in case of rapid recovery of paths dev_loss_tmo 90 # Don't queue traffic down a failed path no_path_retry fail # balance much more aggressively across the active paths rr_min_io 1 The primary goal was to have rapid and reliable failover in a cluster environment (without ping-ponging). The defaults from multipathd gave a 30-90 second pause in I/O every time a path went away - we've managed to get it down to ~5s with the above settings. Note that we've not tried this "in production" yet, but it has held up fine under heavy benchmark load. Along the way we discovered an odd GPFS "feature" - if some nodes in the cluster use RDAC (and thus have /dev/sdXX devices) and some use multipathd (and thus use /dev/dm-XX devices), then the nodes can either fail to find attached NSD devices (in the case of the RDAC host where the NSD's were initially created on a multipath host) or can try to talk to them down the wrong device (for instance - talking to /dev/sdXX rather than /dev/dm-XX). We just set up this mixed environment to compare rdac vs dm-multipath, and don't expect to put it into production - but it's the kind of thing which could end up cropping up in a system migrating from RDAC to dm-multipath, or vice versa. It seems that on creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or "generic" (rdac), and servers using one type can't see the other. We're testing a workaround for the "dm-multipath server accessing via /dev/sdXX" case just now - create the following (executable, root-owned) script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: #!/bin/ksh # # this script ensures that we are not using the raw /dev/sd\* devices for GPFS # but use the multipath /dev/dm-\* devices instead for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) do echo $dev generic done # skip the GPFS device discovery exit 0 except change that simple "$dev generic" echo to one which says "$dev mpp" or "$dev generic" depending on whether the device was created with dm-multipath or rdac attached hosts. The reverse also likely would work to get the rdac host to pick up the dm-multipath created nsd's (echo $dev mpp, for the /dev/sdXX devices). Thankfully, we have no plans to mix the environment - but for future reference it could be important (if ever migrating existing systems from rdac to dm-multipath, for instance). -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From jonathan at buzzard.me.uk Thu Sep 19 16:38:05 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 19 Sep 2013 16:38:05 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523B0FD8.1090403@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> <523B0FD8.1090403@ed.ac.uk> Message-ID: <1379605085.26154.4.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: [SNIP] > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > That is all covered in the GPFS documentation under "don't do that" ;-) [SNIP] > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > The recommendation is roughly umount GPFS everywhere, export the file system, change everything to dm-multipath/RDAC depending on your poison, import file system and mount again. I think you can skip the export/import by doing some fiddling to switch the NSD type. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From cexton26 at gmail.com Sat Sep 21 12:31:16 2013 From: cexton26 at gmail.com (Chris Exton) Date: Sat, 21 Sep 2013 12:31:16 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 20, Issue 3 In-Reply-To: References: Message-ID: I had a similar issue when upgrading from an RDAC based solution to multipath. I found a command that wasn't that well documented to change the device type mmchconfig updateNsdType=/tmp/disks On 20 September 2013 12:00, wrote: > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at gpfsug.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at gpfsug.org > > You can reach the person managing the list at > gpfsug-discuss-owner at gpfsug.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Multipath configurations (Orlando Richards) > 2. Re: Multipath configurations (Jonathan Buzzard) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 19 Sep 2013 15:53:12 +0100 > From: Orlando Richards > To: gpfsug-discuss at gpfsug.org > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <523B0FD8.1090403 at ed.ac.uk> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 16/09/13 16:25, Orlando Richards wrote: > > Hi folks, > > > > We're building a new storage service and are planning on using > > multipathd rather than LSI's rdac to handle the multipathing. > > > > It's all working well, but I'm looking at settling on the final > > parameters for the multipath.conf. In particular, the values for: > > > > * rr_min_io (1?) > > * failback (I think "manual" or "followover"?) > > * no_path_retry (guessing here - fail?) > > * dev_loss_tmo (guessing here - 15?) > > * fast_io_fail_tmo (guessing here - 10?) > > > > Does anyone have a working multipath.conf for LSI based storage systems > > (or others, for that matter), and/or have experience and wisdom to share > > on the above settings (and any others I may have missed?). Any war > > stories about dm-multipath to share? > > > > > > Hi all, > > Thanks for the feedback on all this. From that, and more digging and > testing, we've settled on the following multipath.conf stanzas: > > path_grouping_policy group_by_prio > prio rdac > path_checker rdac > path_selector "round-robin 0" > hardware_handler "1 rdac" > features "2 pg_init_retries 50" > # All "standard" up to here > > # Prevent ping-ponging of controllers, but > # allow for automatic failback > failback followover > # Massively accelerate the failure detection time > # (default settings give ~30-90 seconds, this gives ~5s) > fast_io_fail_tmo 5 > # Keep the /dev device entries in situ for 90 seconds, > # in case of rapid recovery of paths > dev_loss_tmo 90 > # Don't queue traffic down a failed path > no_path_retry fail > # balance much more aggressively across the active paths > rr_min_io 1 > > The primary goal was to have rapid and reliable failover in a cluster > environment (without ping-ponging). The defaults from multipathd gave a > 30-90 second pause in I/O every time a path went away - we've managed to > get it down to ~5s with the above settings. > > Note that we've not tried this "in production" yet, but it has held up > fine under heavy benchmark load. > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > > We're testing a workaround for the "dm-multipath server accessing via > /dev/sdXX" case just now - create the following (executable, root-owned) > script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: > > #!/bin/ksh > # > # this script ensures that we are not using the raw /dev/sd\* devices > for GPFS > # but use the multipath /dev/dm-\* devices instead > for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) > do > echo $dev generic > done > > # skip the GPFS device discovery > exit 0 > > > except change that simple "$dev generic" echo to one which says "$dev > mpp" or "$dev generic" depending on whether the device was created with > dm-multipath or rdac attached hosts. The reverse also likely would work > to get the rdac host to pick up the dm-multipath created nsd's (echo > $dev mpp, for the /dev/sdXX devices). > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > > > > > > > > -- > -- > Dr Orlando Richards > Information Services > IT Infrastructure Division > Unix Section > Tel: 0131 650 4994 > skype: orlando.richards > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > ------------------------------ > > Message: 2 > Date: Thu, 19 Sep 2013 16:38:05 +0100 > From: Jonathan Buzzard > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <1379605085.26154.4.camel at buzzard.phy.strath.ac.uk> > Content-Type: text/plain; charset="UTF-8" > > On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: > > [SNIP] > > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > > cluster use RDAC (and thus have /dev/sdXX devices) and some use > > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > > fail to find attached NSD devices (in the case of the RDAC host where > > the NSD's were initially created on a multipath host) or can try to talk > > to them down the wrong device (for instance - talking to /dev/sdXX > > rather than /dev/dm-XX). We just set up this mixed environment to > > compare rdac vs dm-multipath, and don't expect to put it into production > > - but it's the kind of thing which could end up cropping up in a system > > migrating from RDAC to dm-multipath, or vice versa. It seems that on > > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > > "generic" (rdac), and servers using one type can't see the other. > > > > That is all covered in the GPFS documentation under "don't do that" ;-) > > [SNIP] > > > > > Thankfully, we have no plans to mix the environment - but for future > > reference it could be important (if ever migrating existing systems from > > rdac to dm-multipath, for instance). > > > > The recommendation is roughly umount GPFS everywhere, export the file > system, change everything to dm-multipath/RDAC depending on your poison, > import file system and mount again. I think you can skip the > export/import by doing some fiddling to switch the NSD type. > > > JAB. > > -- > Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk > Fife, United Kingdom. > > > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 20, Issue 3 > ********************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From theo at tepucom.nl Thu Sep 26 20:25:12 2013 From: theo at tepucom.nl (Theo Purmer (Tepucom BV)) Date: Thu, 26 Sep 2013 21:25:12 +0200 (CEST) Subject: [gpfsug-discuss] afm setup Message-ID: <10465077.604006.1380223512803.JavaMail.root@tepucom.nl> Hi Folks We are working on a gpfs poc using afm to sync (async) two remote sites as synchronous syncing is not possible But I do not understand exactly how afm works. I have read all the available documentation (i think) Is there a drawing available of the components of an afm setup or a complete description of the setup. AFM is kinda new, IBM here in the netherlands does not have a tech that knows how it works. Thanks theo -------------- next part -------------- An HTML attachment was scrubbed... URL: From orlando.richards at ed.ac.uk Mon Sep 16 16:25:26 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Mon, 16 Sep 2013 16:25:26 +0100 Subject: [gpfsug-discuss] Multipath configurations Message-ID: <523722E6.3010603@ed.ac.uk> Hi folks, We're building a new storage service and are planning on using multipathd rather than LSI's rdac to handle the multipathing. It's all working well, but I'm looking at settling on the final parameters for the multipath.conf. In particular, the values for: * rr_min_io (1?) * failback (I think "manual" or "followover"?) * no_path_retry (guessing here - fail?) * dev_loss_tmo (guessing here - 15?) * fast_io_fail_tmo (guessing here - 10?) Does anyone have a working multipath.conf for LSI based storage systems (or others, for that matter), and/or have experience and wisdom to share on the above settings (and any others I may have missed?). Any war stories about dm-multipath to share? -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From seanlee at tw.ibm.com Wed Sep 18 17:17:10 2013 From: seanlee at tw.ibm.com (Sean S Lee) Date: Thu, 19 Sep 2013 00:17:10 +0800 Subject: [gpfsug-discuss] Re: Multipath configurations In-Reply-To: References: Message-ID: Hi Orlando Did you read your product documentation? Normally this info can be found under "Working with Multipath.conf file". Some useful info can also be found here: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014889306 Regards, Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From orlando.richards at ed.ac.uk Thu Sep 19 15:53:12 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Thu, 19 Sep 2013 15:53:12 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523722E6.3010603@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> Message-ID: <523B0FD8.1090403@ed.ac.uk> On 16/09/13 16:25, Orlando Richards wrote: > Hi folks, > > We're building a new storage service and are planning on using > multipathd rather than LSI's rdac to handle the multipathing. > > It's all working well, but I'm looking at settling on the final > parameters for the multipath.conf. In particular, the values for: > > * rr_min_io (1?) > * failback (I think "manual" or "followover"?) > * no_path_retry (guessing here - fail?) > * dev_loss_tmo (guessing here - 15?) > * fast_io_fail_tmo (guessing here - 10?) > > Does anyone have a working multipath.conf for LSI based storage systems > (or others, for that matter), and/or have experience and wisdom to share > on the above settings (and any others I may have missed?). Any war > stories about dm-multipath to share? > > Hi all, Thanks for the feedback on all this. From that, and more digging and testing, we've settled on the following multipath.conf stanzas: path_grouping_policy group_by_prio prio rdac path_checker rdac path_selector "round-robin 0" hardware_handler "1 rdac" features "2 pg_init_retries 50" # All "standard" up to here # Prevent ping-ponging of controllers, but # allow for automatic failback failback followover # Massively accelerate the failure detection time # (default settings give ~30-90 seconds, this gives ~5s) fast_io_fail_tmo 5 # Keep the /dev device entries in situ for 90 seconds, # in case of rapid recovery of paths dev_loss_tmo 90 # Don't queue traffic down a failed path no_path_retry fail # balance much more aggressively across the active paths rr_min_io 1 The primary goal was to have rapid and reliable failover in a cluster environment (without ping-ponging). The defaults from multipathd gave a 30-90 second pause in I/O every time a path went away - we've managed to get it down to ~5s with the above settings. Note that we've not tried this "in production" yet, but it has held up fine under heavy benchmark load. Along the way we discovered an odd GPFS "feature" - if some nodes in the cluster use RDAC (and thus have /dev/sdXX devices) and some use multipathd (and thus use /dev/dm-XX devices), then the nodes can either fail to find attached NSD devices (in the case of the RDAC host where the NSD's were initially created on a multipath host) or can try to talk to them down the wrong device (for instance - talking to /dev/sdXX rather than /dev/dm-XX). We just set up this mixed environment to compare rdac vs dm-multipath, and don't expect to put it into production - but it's the kind of thing which could end up cropping up in a system migrating from RDAC to dm-multipath, or vice versa. It seems that on creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or "generic" (rdac), and servers using one type can't see the other. We're testing a workaround for the "dm-multipath server accessing via /dev/sdXX" case just now - create the following (executable, root-owned) script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: #!/bin/ksh # # this script ensures that we are not using the raw /dev/sd\* devices for GPFS # but use the multipath /dev/dm-\* devices instead for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) do echo $dev generic done # skip the GPFS device discovery exit 0 except change that simple "$dev generic" echo to one which says "$dev mpp" or "$dev generic" depending on whether the device was created with dm-multipath or rdac attached hosts. The reverse also likely would work to get the rdac host to pick up the dm-multipath created nsd's (echo $dev mpp, for the /dev/sdXX devices). Thankfully, we have no plans to mix the environment - but for future reference it could be important (if ever migrating existing systems from rdac to dm-multipath, for instance). -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From jonathan at buzzard.me.uk Thu Sep 19 16:38:05 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 19 Sep 2013 16:38:05 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523B0FD8.1090403@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> <523B0FD8.1090403@ed.ac.uk> Message-ID: <1379605085.26154.4.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: [SNIP] > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > That is all covered in the GPFS documentation under "don't do that" ;-) [SNIP] > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > The recommendation is roughly umount GPFS everywhere, export the file system, change everything to dm-multipath/RDAC depending on your poison, import file system and mount again. I think you can skip the export/import by doing some fiddling to switch the NSD type. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From cexton26 at gmail.com Sat Sep 21 12:31:16 2013 From: cexton26 at gmail.com (Chris Exton) Date: Sat, 21 Sep 2013 12:31:16 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 20, Issue 3 In-Reply-To: References: Message-ID: I had a similar issue when upgrading from an RDAC based solution to multipath. I found a command that wasn't that well documented to change the device type mmchconfig updateNsdType=/tmp/disks On 20 September 2013 12:00, wrote: > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at gpfsug.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at gpfsug.org > > You can reach the person managing the list at > gpfsug-discuss-owner at gpfsug.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Multipath configurations (Orlando Richards) > 2. Re: Multipath configurations (Jonathan Buzzard) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 19 Sep 2013 15:53:12 +0100 > From: Orlando Richards > To: gpfsug-discuss at gpfsug.org > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <523B0FD8.1090403 at ed.ac.uk> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 16/09/13 16:25, Orlando Richards wrote: > > Hi folks, > > > > We're building a new storage service and are planning on using > > multipathd rather than LSI's rdac to handle the multipathing. > > > > It's all working well, but I'm looking at settling on the final > > parameters for the multipath.conf. In particular, the values for: > > > > * rr_min_io (1?) > > * failback (I think "manual" or "followover"?) > > * no_path_retry (guessing here - fail?) > > * dev_loss_tmo (guessing here - 15?) > > * fast_io_fail_tmo (guessing here - 10?) > > > > Does anyone have a working multipath.conf for LSI based storage systems > > (or others, for that matter), and/or have experience and wisdom to share > > on the above settings (and any others I may have missed?). Any war > > stories about dm-multipath to share? > > > > > > Hi all, > > Thanks for the feedback on all this. From that, and more digging and > testing, we've settled on the following multipath.conf stanzas: > > path_grouping_policy group_by_prio > prio rdac > path_checker rdac > path_selector "round-robin 0" > hardware_handler "1 rdac" > features "2 pg_init_retries 50" > # All "standard" up to here > > # Prevent ping-ponging of controllers, but > # allow for automatic failback > failback followover > # Massively accelerate the failure detection time > # (default settings give ~30-90 seconds, this gives ~5s) > fast_io_fail_tmo 5 > # Keep the /dev device entries in situ for 90 seconds, > # in case of rapid recovery of paths > dev_loss_tmo 90 > # Don't queue traffic down a failed path > no_path_retry fail > # balance much more aggressively across the active paths > rr_min_io 1 > > The primary goal was to have rapid and reliable failover in a cluster > environment (without ping-ponging). The defaults from multipathd gave a > 30-90 second pause in I/O every time a path went away - we've managed to > get it down to ~5s with the above settings. > > Note that we've not tried this "in production" yet, but it has held up > fine under heavy benchmark load. > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > > We're testing a workaround for the "dm-multipath server accessing via > /dev/sdXX" case just now - create the following (executable, root-owned) > script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: > > #!/bin/ksh > # > # this script ensures that we are not using the raw /dev/sd\* devices > for GPFS > # but use the multipath /dev/dm-\* devices instead > for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) > do > echo $dev generic > done > > # skip the GPFS device discovery > exit 0 > > > except change that simple "$dev generic" echo to one which says "$dev > mpp" or "$dev generic" depending on whether the device was created with > dm-multipath or rdac attached hosts. The reverse also likely would work > to get the rdac host to pick up the dm-multipath created nsd's (echo > $dev mpp, for the /dev/sdXX devices). > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > > > > > > > > -- > -- > Dr Orlando Richards > Information Services > IT Infrastructure Division > Unix Section > Tel: 0131 650 4994 > skype: orlando.richards > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > ------------------------------ > > Message: 2 > Date: Thu, 19 Sep 2013 16:38:05 +0100 > From: Jonathan Buzzard > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <1379605085.26154.4.camel at buzzard.phy.strath.ac.uk> > Content-Type: text/plain; charset="UTF-8" > > On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: > > [SNIP] > > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > > cluster use RDAC (and thus have /dev/sdXX devices) and some use > > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > > fail to find attached NSD devices (in the case of the RDAC host where > > the NSD's were initially created on a multipath host) or can try to talk > > to them down the wrong device (for instance - talking to /dev/sdXX > > rather than /dev/dm-XX). We just set up this mixed environment to > > compare rdac vs dm-multipath, and don't expect to put it into production > > - but it's the kind of thing which could end up cropping up in a system > > migrating from RDAC to dm-multipath, or vice versa. It seems that on > > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > > "generic" (rdac), and servers using one type can't see the other. > > > > That is all covered in the GPFS documentation under "don't do that" ;-) > > [SNIP] > > > > > Thankfully, we have no plans to mix the environment - but for future > > reference it could be important (if ever migrating existing systems from > > rdac to dm-multipath, for instance). > > > > The recommendation is roughly umount GPFS everywhere, export the file > system, change everything to dm-multipath/RDAC depending on your poison, > import file system and mount again. I think you can skip the > export/import by doing some fiddling to switch the NSD type. > > > JAB. > > -- > Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk > Fife, United Kingdom. > > > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 20, Issue 3 > ********************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From theo at tepucom.nl Thu Sep 26 20:25:12 2013 From: theo at tepucom.nl (Theo Purmer (Tepucom BV)) Date: Thu, 26 Sep 2013 21:25:12 +0200 (CEST) Subject: [gpfsug-discuss] afm setup Message-ID: <10465077.604006.1380223512803.JavaMail.root@tepucom.nl> Hi Folks We are working on a gpfs poc using afm to sync (async) two remote sites as synchronous syncing is not possible But I do not understand exactly how afm works. I have read all the available documentation (i think) Is there a drawing available of the components of an afm setup or a complete description of the setup. AFM is kinda new, IBM here in the netherlands does not have a tech that knows how it works. Thanks theo -------------- next part -------------- An HTML attachment was scrubbed... URL: From orlando.richards at ed.ac.uk Mon Sep 16 16:25:26 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Mon, 16 Sep 2013 16:25:26 +0100 Subject: [gpfsug-discuss] Multipath configurations Message-ID: <523722E6.3010603@ed.ac.uk> Hi folks, We're building a new storage service and are planning on using multipathd rather than LSI's rdac to handle the multipathing. It's all working well, but I'm looking at settling on the final parameters for the multipath.conf. In particular, the values for: * rr_min_io (1?) * failback (I think "manual" or "followover"?) * no_path_retry (guessing here - fail?) * dev_loss_tmo (guessing here - 15?) * fast_io_fail_tmo (guessing here - 10?) Does anyone have a working multipath.conf for LSI based storage systems (or others, for that matter), and/or have experience and wisdom to share on the above settings (and any others I may have missed?). Any war stories about dm-multipath to share? -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From seanlee at tw.ibm.com Wed Sep 18 17:17:10 2013 From: seanlee at tw.ibm.com (Sean S Lee) Date: Thu, 19 Sep 2013 00:17:10 +0800 Subject: [gpfsug-discuss] Re: Multipath configurations In-Reply-To: References: Message-ID: Hi Orlando Did you read your product documentation? Normally this info can be found under "Working with Multipath.conf file". Some useful info can also be found here: https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014889306 Regards, Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From orlando.richards at ed.ac.uk Thu Sep 19 15:53:12 2013 From: orlando.richards at ed.ac.uk (Orlando Richards) Date: Thu, 19 Sep 2013 15:53:12 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523722E6.3010603@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> Message-ID: <523B0FD8.1090403@ed.ac.uk> On 16/09/13 16:25, Orlando Richards wrote: > Hi folks, > > We're building a new storage service and are planning on using > multipathd rather than LSI's rdac to handle the multipathing. > > It's all working well, but I'm looking at settling on the final > parameters for the multipath.conf. In particular, the values for: > > * rr_min_io (1?) > * failback (I think "manual" or "followover"?) > * no_path_retry (guessing here - fail?) > * dev_loss_tmo (guessing here - 15?) > * fast_io_fail_tmo (guessing here - 10?) > > Does anyone have a working multipath.conf for LSI based storage systems > (or others, for that matter), and/or have experience and wisdom to share > on the above settings (and any others I may have missed?). Any war > stories about dm-multipath to share? > > Hi all, Thanks for the feedback on all this. From that, and more digging and testing, we've settled on the following multipath.conf stanzas: path_grouping_policy group_by_prio prio rdac path_checker rdac path_selector "round-robin 0" hardware_handler "1 rdac" features "2 pg_init_retries 50" # All "standard" up to here # Prevent ping-ponging of controllers, but # allow for automatic failback failback followover # Massively accelerate the failure detection time # (default settings give ~30-90 seconds, this gives ~5s) fast_io_fail_tmo 5 # Keep the /dev device entries in situ for 90 seconds, # in case of rapid recovery of paths dev_loss_tmo 90 # Don't queue traffic down a failed path no_path_retry fail # balance much more aggressively across the active paths rr_min_io 1 The primary goal was to have rapid and reliable failover in a cluster environment (without ping-ponging). The defaults from multipathd gave a 30-90 second pause in I/O every time a path went away - we've managed to get it down to ~5s with the above settings. Note that we've not tried this "in production" yet, but it has held up fine under heavy benchmark load. Along the way we discovered an odd GPFS "feature" - if some nodes in the cluster use RDAC (and thus have /dev/sdXX devices) and some use multipathd (and thus use /dev/dm-XX devices), then the nodes can either fail to find attached NSD devices (in the case of the RDAC host where the NSD's were initially created on a multipath host) or can try to talk to them down the wrong device (for instance - talking to /dev/sdXX rather than /dev/dm-XX). We just set up this mixed environment to compare rdac vs dm-multipath, and don't expect to put it into production - but it's the kind of thing which could end up cropping up in a system migrating from RDAC to dm-multipath, or vice versa. It seems that on creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or "generic" (rdac), and servers using one type can't see the other. We're testing a workaround for the "dm-multipath server accessing via /dev/sdXX" case just now - create the following (executable, root-owned) script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: #!/bin/ksh # # this script ensures that we are not using the raw /dev/sd\* devices for GPFS # but use the multipath /dev/dm-\* devices instead for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) do echo $dev generic done # skip the GPFS device discovery exit 0 except change that simple "$dev generic" echo to one which says "$dev mpp" or "$dev generic" depending on whether the device was created with dm-multipath or rdac attached hosts. The reverse also likely would work to get the rdac host to pick up the dm-multipath created nsd's (echo $dev mpp, for the /dev/sdXX devices). Thankfully, we have no plans to mix the environment - but for future reference it could be important (if ever migrating existing systems from rdac to dm-multipath, for instance). -- -- Dr Orlando Richards Information Services IT Infrastructure Division Unix Section Tel: 0131 650 4994 skype: orlando.richards The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. From jonathan at buzzard.me.uk Thu Sep 19 16:38:05 2013 From: jonathan at buzzard.me.uk (Jonathan Buzzard) Date: Thu, 19 Sep 2013 16:38:05 +0100 Subject: [gpfsug-discuss] Multipath configurations In-Reply-To: <523B0FD8.1090403@ed.ac.uk> References: <523722E6.3010603@ed.ac.uk> <523B0FD8.1090403@ed.ac.uk> Message-ID: <1379605085.26154.4.camel@buzzard.phy.strath.ac.uk> On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: [SNIP] > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > That is all covered in the GPFS documentation under "don't do that" ;-) [SNIP] > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > The recommendation is roughly umount GPFS everywhere, export the file system, change everything to dm-multipath/RDAC depending on your poison, import file system and mount again. I think you can skip the export/import by doing some fiddling to switch the NSD type. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. From cexton26 at gmail.com Sat Sep 21 12:31:16 2013 From: cexton26 at gmail.com (Chris Exton) Date: Sat, 21 Sep 2013 12:31:16 +0100 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 20, Issue 3 In-Reply-To: References: Message-ID: I had a similar issue when upgrading from an RDAC based solution to multipath. I found a command that wasn't that well documented to change the device type mmchconfig updateNsdType=/tmp/disks On 20 September 2013 12:00, wrote: > Send gpfsug-discuss mailing list submissions to > gpfsug-discuss at gpfsug.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > or, via email, send a message with subject or body 'help' to > gpfsug-discuss-request at gpfsug.org > > You can reach the person managing the list at > gpfsug-discuss-owner at gpfsug.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > 1. Re: Multipath configurations (Orlando Richards) > 2. Re: Multipath configurations (Jonathan Buzzard) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 19 Sep 2013 15:53:12 +0100 > From: Orlando Richards > To: gpfsug-discuss at gpfsug.org > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <523B0FD8.1090403 at ed.ac.uk> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 16/09/13 16:25, Orlando Richards wrote: > > Hi folks, > > > > We're building a new storage service and are planning on using > > multipathd rather than LSI's rdac to handle the multipathing. > > > > It's all working well, but I'm looking at settling on the final > > parameters for the multipath.conf. In particular, the values for: > > > > * rr_min_io (1?) > > * failback (I think "manual" or "followover"?) > > * no_path_retry (guessing here - fail?) > > * dev_loss_tmo (guessing here - 15?) > > * fast_io_fail_tmo (guessing here - 10?) > > > > Does anyone have a working multipath.conf for LSI based storage systems > > (or others, for that matter), and/or have experience and wisdom to share > > on the above settings (and any others I may have missed?). Any war > > stories about dm-multipath to share? > > > > > > Hi all, > > Thanks for the feedback on all this. From that, and more digging and > testing, we've settled on the following multipath.conf stanzas: > > path_grouping_policy group_by_prio > prio rdac > path_checker rdac > path_selector "round-robin 0" > hardware_handler "1 rdac" > features "2 pg_init_retries 50" > # All "standard" up to here > > # Prevent ping-ponging of controllers, but > # allow for automatic failback > failback followover > # Massively accelerate the failure detection time > # (default settings give ~30-90 seconds, this gives ~5s) > fast_io_fail_tmo 5 > # Keep the /dev device entries in situ for 90 seconds, > # in case of rapid recovery of paths > dev_loss_tmo 90 > # Don't queue traffic down a failed path > no_path_retry fail > # balance much more aggressively across the active paths > rr_min_io 1 > > The primary goal was to have rapid and reliable failover in a cluster > environment (without ping-ponging). The defaults from multipathd gave a > 30-90 second pause in I/O every time a path went away - we've managed to > get it down to ~5s with the above settings. > > Note that we've not tried this "in production" yet, but it has held up > fine under heavy benchmark load. > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > cluster use RDAC (and thus have /dev/sdXX devices) and some use > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > fail to find attached NSD devices (in the case of the RDAC host where > the NSD's were initially created on a multipath host) or can try to talk > to them down the wrong device (for instance - talking to /dev/sdXX > rather than /dev/dm-XX). We just set up this mixed environment to > compare rdac vs dm-multipath, and don't expect to put it into production > - but it's the kind of thing which could end up cropping up in a system > migrating from RDAC to dm-multipath, or vice versa. It seems that on > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > "generic" (rdac), and servers using one type can't see the other. > > We're testing a workaround for the "dm-multipath server accessing via > /dev/sdXX" case just now - create the following (executable, root-owned) > script in /var/mmfs/etc/nsddevices on the dm-multipath hosts: > > #!/bin/ksh > # > # this script ensures that we are not using the raw /dev/sd\* devices > for GPFS > # but use the multipath /dev/dm-\* devices instead > for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' ) > do > echo $dev generic > done > > # skip the GPFS device discovery > exit 0 > > > except change that simple "$dev generic" echo to one which says "$dev > mpp" or "$dev generic" depending on whether the device was created with > dm-multipath or rdac attached hosts. The reverse also likely would work > to get the rdac host to pick up the dm-multipath created nsd's (echo > $dev mpp, for the /dev/sdXX devices). > > Thankfully, we have no plans to mix the environment - but for future > reference it could be important (if ever migrating existing systems from > rdac to dm-multipath, for instance). > > > > > > > > -- > -- > Dr Orlando Richards > Information Services > IT Infrastructure Division > Unix Section > Tel: 0131 650 4994 > skype: orlando.richards > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > ------------------------------ > > Message: 2 > Date: Thu, 19 Sep 2013 16:38:05 +0100 > From: Jonathan Buzzard > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] Multipath configurations > Message-ID: <1379605085.26154.4.camel at buzzard.phy.strath.ac.uk> > Content-Type: text/plain; charset="UTF-8" > > On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote: > > [SNIP] > > > Along the way we discovered an odd GPFS "feature" - if some nodes in the > > cluster use RDAC (and thus have /dev/sdXX devices) and some use > > multipathd (and thus use /dev/dm-XX devices), then the nodes can either > > fail to find attached NSD devices (in the case of the RDAC host where > > the NSD's were initially created on a multipath host) or can try to talk > > to them down the wrong device (for instance - talking to /dev/sdXX > > rather than /dev/dm-XX). We just set up this mixed environment to > > compare rdac vs dm-multipath, and don't expect to put it into production > > - but it's the kind of thing which could end up cropping up in a system > > migrating from RDAC to dm-multipath, or vice versa. It seems that on > > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or > > "generic" (rdac), and servers using one type can't see the other. > > > > That is all covered in the GPFS documentation under "don't do that" ;-) > > [SNIP] > > > > > Thankfully, we have no plans to mix the environment - but for future > > reference it could be important (if ever migrating existing systems from > > rdac to dm-multipath, for instance). > > > > The recommendation is roughly umount GPFS everywhere, export the file > system, change everything to dm-multipath/RDAC depending on your poison, > import file system and mount again. I think you can skip the > export/import by doing some fiddling to switch the NSD type. > > > JAB. > > -- > Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk > Fife, United Kingdom. > > > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at gpfsug.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > End of gpfsug-discuss Digest, Vol 20, Issue 3 > ********************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From theo at tepucom.nl Thu Sep 26 20:25:12 2013 From: theo at tepucom.nl (Theo Purmer (Tepucom BV)) Date: Thu, 26 Sep 2013 21:25:12 +0200 (CEST) Subject: [gpfsug-discuss] afm setup Message-ID: <10465077.604006.1380223512803.JavaMail.root@tepucom.nl> Hi Folks We are working on a gpfs poc using afm to sync (async) two remote sites as synchronous syncing is not possible But I do not understand exactly how afm works. I have read all the available documentation (i think) Is there a drawing available of the components of an afm setup or a complete description of the setup. AFM is kinda new, IBM here in the netherlands does not have a tech that knows how it works. Thanks theo -------------- next part -------------- An HTML attachment was scrubbed... URL: