[gpfsug-discuss] gpfsug-discuss Digest, Vol 20, Issue 3
Chris Exton
cexton26 at gmail.com
Sat Sep 21 12:31:16 BST 2013
I had a similar issue when upgrading from an RDAC based solution to
multipath.
I found a command that wasn't that well documented to change the device
type mmchconfig updateNsdType=/tmp/disks
On 20 September 2013 12:00, <gpfsug-discuss-request at gpfsug.org> wrote:
> Send gpfsug-discuss mailing list submissions to
> gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
> gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
> gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
> 1. Re: Multipath configurations (Orlando Richards)
> 2. Re: Multipath configurations (Jonathan Buzzard)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 19 Sep 2013 15:53:12 +0100
> From: Orlando Richards <orlando.richards at ed.ac.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] Multipath configurations
> Message-ID: <523B0FD8.1090403 at ed.ac.uk>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> On 16/09/13 16:25, Orlando Richards wrote:
> > Hi folks,
> >
> > We're building a new storage service and are planning on using
> > multipathd rather than LSI's rdac to handle the multipathing.
> >
> > It's all working well, but I'm looking at settling on the final
> > parameters for the multipath.conf. In particular, the values for:
> >
> > * rr_min_io (1?)
> > * failback (I think "manual" or "followover"?)
> > * no_path_retry (guessing here - fail?)
> > * dev_loss_tmo (guessing here - 15?)
> > * fast_io_fail_tmo (guessing here - 10?)
> >
> > Does anyone have a working multipath.conf for LSI based storage systems
> > (or others, for that matter), and/or have experience and wisdom to share
> > on the above settings (and any others I may have missed?). Any war
> > stories about dm-multipath to share?
> >
> >
>
> Hi all,
>
> Thanks for the feedback on all this. From that, and more digging and
> testing, we've settled on the following multipath.conf stanzas:
>
> path_grouping_policy group_by_prio
> prio rdac
> path_checker rdac
> path_selector "round-robin 0"
> hardware_handler "1 rdac"
> features "2 pg_init_retries 50"
> # All "standard" up to here
>
> # Prevent ping-ponging of controllers, but
> # allow for automatic failback
> failback followover
> # Massively accelerate the failure detection time
> # (default settings give ~30-90 seconds, this gives ~5s)
> fast_io_fail_tmo 5
> # Keep the /dev device entries in situ for 90 seconds,
> # in case of rapid recovery of paths
> dev_loss_tmo 90
> # Don't queue traffic down a failed path
> no_path_retry fail
> # balance much more aggressively across the active paths
> rr_min_io 1
>
> The primary goal was to have rapid and reliable failover in a cluster
> environment (without ping-ponging). The defaults from multipathd gave a
> 30-90 second pause in I/O every time a path went away - we've managed to
> get it down to ~5s with the above settings.
>
> Note that we've not tried this "in production" yet, but it has held up
> fine under heavy benchmark load.
>
> Along the way we discovered an odd GPFS "feature" - if some nodes in the
> cluster use RDAC (and thus have /dev/sdXX devices) and some use
> multipathd (and thus use /dev/dm-XX devices), then the nodes can either
> fail to find attached NSD devices (in the case of the RDAC host where
> the NSD's were initially created on a multipath host) or can try to talk
> to them down the wrong device (for instance - talking to /dev/sdXX
> rather than /dev/dm-XX). We just set up this mixed environment to
> compare rdac vs dm-multipath, and don't expect to put it into production
> - but it's the kind of thing which could end up cropping up in a system
> migrating from RDAC to dm-multipath, or vice versa. It seems that on
> creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or
> "generic" (rdac), and servers using one type can't see the other.
>
> We're testing a workaround for the "dm-multipath server accessing via
> /dev/sdXX" case just now - create the following (executable, root-owned)
> script in /var/mmfs/etc/nsddevices on the dm-multipath hosts:
>
> #!/bin/ksh
> #
> # this script ensures that we are not using the raw /dev/sd\* devices
> for GPFS
> # but use the multipath /dev/dm-\* devices instead
> for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' )
> do
> echo $dev generic
> done
>
> # skip the GPFS device discovery
> exit 0
>
>
> except change that simple "$dev generic" echo to one which says "$dev
> mpp" or "$dev generic" depending on whether the device was created with
> dm-multipath or rdac attached hosts. The reverse also likely would work
> to get the rdac host to pick up the dm-multipath created nsd's (echo
> $dev mpp, for the /dev/sdXX devices).
>
> Thankfully, we have no plans to mix the environment - but for future
> reference it could be important (if ever migrating existing systems from
> rdac to dm-multipath, for instance).
>
>
>
>
>
>
>
> --
> --
> Dr Orlando Richards
> Information Services
> IT Infrastructure Division
> Unix Section
> Tel: 0131 650 4994
> skype: orlando.richards
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 19 Sep 2013 16:38:05 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: Re: [gpfsug-discuss] Multipath configurations
> Message-ID: <1379605085.26154.4.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote:
>
> [SNIP]
>
> > Along the way we discovered an odd GPFS "feature" - if some nodes in the
> > cluster use RDAC (and thus have /dev/sdXX devices) and some use
> > multipathd (and thus use /dev/dm-XX devices), then the nodes can either
> > fail to find attached NSD devices (in the case of the RDAC host where
> > the NSD's were initially created on a multipath host) or can try to talk
> > to them down the wrong device (for instance - talking to /dev/sdXX
> > rather than /dev/dm-XX). We just set up this mixed environment to
> > compare rdac vs dm-multipath, and don't expect to put it into production
> > - but it's the kind of thing which could end up cropping up in a system
> > migrating from RDAC to dm-multipath, or vice versa. It seems that on
> > creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or
> > "generic" (rdac), and servers using one type can't see the other.
> >
>
> That is all covered in the GPFS documentation under "don't do that" ;-)
>
> [SNIP]
>
> >
> > Thankfully, we have no plans to mix the environment - but for future
> > reference it could be important (if ever migrating existing systems from
> > rdac to dm-multipath, for instance).
> >
>
> The recommendation is roughly umount GPFS everywhere, export the file
> system, change everything to dm-multipath/RDAC depending on your poison,
> import file system and mount again. I think you can skip the
> export/import by doing some fiddling to switch the NSD type.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 20, Issue 3
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20130921/b4fc3454/attachment-0002.htm>
More information about the gpfsug-discuss
mailing list