<div dir="ltr">I had a similar issue when upgrading from an RDAC based solution to multipath.<div><br></div><div>I found a command that wasn't that well documented to change the device type mmchconfig <span style="font-family:Arial,sans-serif;font-size:12px;line-height:18px">updateNsdType=/tmp/disks</span></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On 20 September 2013 12:00, <span dir="ltr"><<a href="mailto:gpfsug-discuss-request@gpfsug.org" target="_blank">gpfsug-discuss-request@gpfsug.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send gpfsug-discuss mailing list submissions to<br>
<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:gpfsug-discuss-request@gpfsug.org">gpfsug-discuss-request@gpfsug.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:gpfsug-discuss-owner@gpfsug.org">gpfsug-discuss-owner@gpfsug.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gpfsug-discuss digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Multipath configurations (Orlando Richards)<br>
2. Re: Multipath configurations (Jonathan Buzzard)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Thu, 19 Sep 2013 15:53:12 +0100<br>
From: Orlando Richards <<a href="mailto:orlando.richards@ed.ac.uk">orlando.richards@ed.ac.uk</a>><br>
To: <a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a><br>
Subject: Re: [gpfsug-discuss] Multipath configurations<br>
Message-ID: <<a href="mailto:523B0FD8.1090403@ed.ac.uk">523B0FD8.1090403@ed.ac.uk</a>><br>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br>
<br>
On 16/09/13 16:25, Orlando Richards wrote:<br>
> Hi folks,<br>
><br>
> We're building a new storage service and are planning on using<br>
> multipathd rather than LSI's rdac to handle the multipathing.<br>
><br>
> It's all working well, but I'm looking at settling on the final<br>
> parameters for the multipath.conf. In particular, the values for:<br>
><br>
> * rr_min_io (1?)<br>
> * failback (I think "manual" or "followover"?)<br>
> * no_path_retry (guessing here - fail?)<br>
> * dev_loss_tmo (guessing here - 15?)<br>
> * fast_io_fail_tmo (guessing here - 10?)<br>
><br>
> Does anyone have a working multipath.conf for LSI based storage systems<br>
> (or others, for that matter), and/or have experience and wisdom to share<br>
> on the above settings (and any others I may have missed?). Any war<br>
> stories about dm-multipath to share?<br>
><br>
><br>
<br>
Hi all,<br>
<br>
Thanks for the feedback on all this. From that, and more digging and<br>
testing, we've settled on the following multipath.conf stanzas:<br>
<br>
path_grouping_policy group_by_prio<br>
prio rdac<br>
path_checker rdac<br>
path_selector "round-robin 0"<br>
hardware_handler "1 rdac"<br>
features "2 pg_init_retries 50"<br>
# All "standard" up to here<br>
<br>
# Prevent ping-ponging of controllers, but<br>
# allow for automatic failback<br>
failback followover<br>
# Massively accelerate the failure detection time<br>
# (default settings give ~30-90 seconds, this gives ~5s)<br>
fast_io_fail_tmo 5<br>
# Keep the /dev device entries in situ for 90 seconds,<br>
# in case of rapid recovery of paths<br>
dev_loss_tmo 90<br>
# Don't queue traffic down a failed path<br>
no_path_retry fail<br>
# balance much more aggressively across the active paths<br>
rr_min_io 1<br>
<br>
The primary goal was to have rapid and reliable failover in a cluster<br>
environment (without ping-ponging). The defaults from multipathd gave a<br>
30-90 second pause in I/O every time a path went away - we've managed to<br>
get it down to ~5s with the above settings.<br>
<br>
Note that we've not tried this "in production" yet, but it has held up<br>
fine under heavy benchmark load.<br>
<br>
Along the way we discovered an odd GPFS "feature" - if some nodes in the<br>
cluster use RDAC (and thus have /dev/sdXX devices) and some use<br>
multipathd (and thus use /dev/dm-XX devices), then the nodes can either<br>
fail to find attached NSD devices (in the case of the RDAC host where<br>
the NSD's were initially created on a multipath host) or can try to talk<br>
to them down the wrong device (for instance - talking to /dev/sdXX<br>
rather than /dev/dm-XX). We just set up this mixed environment to<br>
compare rdac vs dm-multipath, and don't expect to put it into production<br>
- but it's the kind of thing which could end up cropping up in a system<br>
migrating from RDAC to dm-multipath, or vice versa. It seems that on<br>
creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or<br>
"generic" (rdac), and servers using one type can't see the other.<br>
<br>
We're testing a workaround for the "dm-multipath server accessing via<br>
/dev/sdXX" case just now - create the following (executable, root-owned)<br>
script in /var/mmfs/etc/nsddevices on the dm-multipath hosts:<br>
<br>
#!/bin/ksh<br>
#<br>
# this script ensures that we are not using the raw /dev/sd\* devices<br>
for GPFS<br>
# but use the multipath /dev/dm-\* devices instead<br>
for dev in $( cat /proc/partitions | grep dm- | awk '{print $4}' )<br>
do<br>
echo $dev generic<br>
done<br>
<br>
# skip the GPFS device discovery<br>
exit 0<br>
<br>
<br>
except change that simple "$dev generic" echo to one which says "$dev<br>
mpp" or "$dev generic" depending on whether the device was created with<br>
dm-multipath or rdac attached hosts. The reverse also likely would work<br>
to get the rdac host to pick up the dm-multipath created nsd's (echo<br>
$dev mpp, for the /dev/sdXX devices).<br>
<br>
Thankfully, we have no plans to mix the environment - but for future<br>
reference it could be important (if ever migrating existing systems from<br>
rdac to dm-multipath, for instance).<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
--<br>
--<br>
Dr Orlando Richards<br>
Information Services<br>
IT Infrastructure Division<br>
Unix Section<br>
Tel: 0131 650 4994<br>
skype: orlando.richards<br>
<br>
The University of Edinburgh is a charitable body, registered in<br>
Scotland, with registration number SC005336.<br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Thu, 19 Sep 2013 16:38:05 +0100<br>
From: Jonathan Buzzard <<a href="mailto:jonathan@buzzard.me.uk">jonathan@buzzard.me.uk</a>><br>
To: gpfsug main discussion list <<a href="mailto:gpfsug-discuss@gpfsug.org">gpfsug-discuss@gpfsug.org</a>><br>
Subject: Re: [gpfsug-discuss] Multipath configurations<br>
Message-ID: <<a href="mailto:1379605085.26154.4.camel@buzzard.phy.strath.ac.uk">1379605085.26154.4.camel@buzzard.phy.strath.ac.uk</a>><br>
Content-Type: text/plain; charset="UTF-8"<br>
<br>
On Thu, 2013-09-19 at 15:53 +0100, Orlando Richards wrote:<br>
<br>
[SNIP]<br>
<br>
> Along the way we discovered an odd GPFS "feature" - if some nodes in the<br>
> cluster use RDAC (and thus have /dev/sdXX devices) and some use<br>
> multipathd (and thus use /dev/dm-XX devices), then the nodes can either<br>
> fail to find attached NSD devices (in the case of the RDAC host where<br>
> the NSD's were initially created on a multipath host) or can try to talk<br>
> to them down the wrong device (for instance - talking to /dev/sdXX<br>
> rather than /dev/dm-XX). We just set up this mixed environment to<br>
> compare rdac vs dm-multipath, and don't expect to put it into production<br>
> - but it's the kind of thing which could end up cropping up in a system<br>
> migrating from RDAC to dm-multipath, or vice versa. It seems that on<br>
> creation, the nsd is tagged somewhere as either "dmm" (dm-multipath) or<br>
> "generic" (rdac), and servers using one type can't see the other.<br>
><br>
<br>
That is all covered in the GPFS documentation under "don't do that" ;-)<br>
<br>
[SNIP]<br>
<br>
><br>
> Thankfully, we have no plans to mix the environment - but for future<br>
> reference it could be important (if ever migrating existing systems from<br>
> rdac to dm-multipath, for instance).<br>
><br>
<br>
The recommendation is roughly umount GPFS everywhere, export the file<br>
system, change everything to dm-multipath/RDAC depending on your poison,<br>
import file system and mount again. I think you can skip the<br>
export/import by doing some fiddling to switch the NSD type.<br>
<br>
<br>
JAB.<br>
<br>
--<br>
Jonathan A. Buzzard Email: jonathan (at) <a href="http://buzzard.me.uk" target="_blank">buzzard.me.uk</a><br>
Fife, United Kingdom.<br>
<br>
<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 20, Issue 3<br>
*********************************************<br>
</blockquote></div><br></div>