[gpfsug-discuss] translating /dev device into nsd name
david_johnson at brown.edu
david_johnson at brown.edu
Mon Dec 19 17:16:07 GMT 2016
We have each of our NSDs on boxes shared between two servers, with one server primary for each raid unit.
When I create Logical drives and map them, I make sure there is no overlap in the logical unit numbers between the two boxes. Then I use /proc/partitions and lsscsi to see if they all show up. When it is time to write the stanza files, I use multipath -ll to get a list with the device name and LUN info, and sort it to round robin over all the NSD servers. It's still tedious, but it doesn't require a trip to the machine room.
Note that the multipath -ll command needs to be run separately on each NSD server to get the device name specific to that host -- the first server name in the list.
Also realize that leaving the host name off when creating NSDs only works if all the drives are visible from the node where you run the command.
Regards,
-- ddj
Dave Johnson
> On Dec 19, 2016, at 8:43 AM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:
>
> Hi Stephen,
>
> Right - that’s what I meant by having the proper device name for the NSD from the NSD server you want to be primary for it. Thanks for confirming that for me.
>
> This discussion prompts me to throw out a related question that will in all likelihood be impossible to answer since it is hardware dependent, AFAIK. But in case I’m wrong about that, I’ll ask. ;-)
>
> My method for identifying the correct “/dev” device to pass to mmcrnsd has been to:
>
> 1. go down to the data center and sit in front of the storage arrays.
> 2. log on to the NSD server I want to be primary for a given NSD.
> 2. use “fdisk -l” to get a list of the disks the NSD server sees and eliminate any that don’t match with the size of the NSD(s) being added.
> 3. for the remaining disks, run “dd if=/dev/<whatever of=/dev/null bs=512k count=1000” against them one at a time and watch to see if the lights for the NSD I’m interested in start blinking.
>
> Is there a better way? Thanks...
>
> Kevin
>
>> On Dec 19, 2016, at 10:16 AM, Stephen Ulmer <ulmer at ulmer.org> wrote:
>>
>> Your observation is correct! There’s usually another step, though:
>>
>> mmcrnsd creates each NSD on the first server in the list, so if you “stripe” the servers you have to know the device name for that NSD on the node that is first in the server list for that NSD. It is usually less work to pick one node, create the NSDs and then change them to have a different server order.
>>
>> --
>> Stephen
>>
>>
>>
>>> On Dec 19, 2016, at 10:58 AM, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:
>>>
>>> Hi Ken,
>>>
>>> Umm, wouldn’t that make that server the primary NSD server for all those NSDs? Granted, you run the mmcrnsd command from one arbitrarily chosen server, but as long as you have the proper device name for the NSD from the NSD server you want to be primary for it, I’ve never had a problem specifying many different servers first in the list.
>>>
>>> Or am I completely misunderstanding what you’re saying? Thanks...
>>>
>>> Kevin
>>>
>>>> On Dec 19, 2016, at 9:30 AM, Ken Hill <kenh at us.ibm.com> wrote:
>>>>
>>>> Indeed. It only matters when deploying NSDs. Post-deployment, all luns (NSDs) are labeled - and they are assembled by GPFS.
>>>>
>>>> Keep in mind: If you are deploying multiple NSDs (with multiple servers) - you'll need to pick one server to work with... Use that server to label the luns (mmcrnsd)... In the nsd stanza file - the server you choose will need to be the first server in the "servers" list.
>>>>
>>>>
>>>> Ken Hill
>>>> Technical Sales Specialist | Software Defined Solution Sales
>>>> IBM Systems
>>>> Phone:1-540-207-7270
>>>> E-mail: kenh at us.ibm.com
>>>> <ATT00001.png> <ATT00002.png> <ATT00003.png> <ATT00004.png> <ATT00005.png> <ATT00006.png> <ATT00007.png> <ATT00008.png> <ATT00009.png> <ATT00010.png> <ATT00011.png>
>>>>
>>>> 2300 Dulles Station Blvd
>>>> Herndon, VA 20171-6133
>>>> United States
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: "Daniel Kidger" <daniel.kidger at uk.ibm.com>
>>>> To: "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
>>>> Cc: "gpfsug main discussion list" <gpfsug-discuss at spectrumscale.org>
>>>> Date: 12/19/2016 06:42 AM
>>>> Subject: Re: [gpfsug-discuss] translating /dev device into nsd name
>>>> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>>>>
>>>>
>>>>
>>>> Valdis wrote:
>>>> Keep in mind that if you have multiple NSD servers in the cluster, there
>>>> is *no* guarantee that the names for a device will be consistent across
>>>> the servers, or across reboots. And when multipath is involved, you may
>>>> have 4 or 8 or even more names for the same device....
>>>>
>>>> Indeed the is whole greatness about NSDs (and in passing why Lustre can be much more tricky to safely manage.)
>>>> Once a lun is "labelled" as an NSD then that NSD name is all you need to care about as the /dev entries can now freely change on reboot or differ across nodes. Indeed if you connect an arbitrary node to an NSD disk via a SAN cable, gpfs will recognise it and use it as a shortcut to that lun.
>>>>
>>>> Finally recall that in the NSD stanza file the /dev entry is only matched for on the first of the listed NSD servers; the other NSD servers will discover and learn which NSD this is, ignoring the /dev value in this stanza.
>>>>
>>>> Daniel
>>>>
>>>> IBM Spectrum Storage Software
>>>> +44 (0)7818 522266
>>>> Sent from my iPad using IBM Verse
>>>>
>>>>
>>>> On 17 Dec 2016, 21:43:00, Valdis.Kletnieks at vt.edu wrote:
>>>>
>>>> From: Valdis.Kletnieks at vt.edu
>>>> To: gpfsug-discuss at spectrumscale.org
>>>> Cc:
>>>> Date: 17 Dec 2016 21:43:00
>>>> Subject: Re: [gpfsug-discuss] translating /dev device into nsd name
>>>>
>>>> On Fri, 16 Dec 2016 23:24:34 -0500, Aaron Knister said:
>>>> > that I can then parse and map the nsd id to the nsd name. I hesitate
>>>> > calling ts* commands directly and I admit it's perhaps an irrational
>>>> > fear, but I associate the -D flag with "delete" in my head and am afraid
>>>> > that some day -D may be just that and *poof* there go my NSD descriptors.
>>>> Others have mentioned mmlsdnsd -m and -X
>>>> Keep in mind that if you have multiple NSD servers in the cluster, there
>>>> is *no* guarantee that the names for a device will be consistent across
>>>> the servers, or across reboots. And when multipath is involved, you may
>>>> have 4 or 8 or even more names for the same device....
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org
>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>> Unless stated otherwise above:
>>>> IBM United Kingdom Limited - Registered in England and Wales with number 741598.
>>>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org
>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org
>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20161219/52465836/attachment-0002.htm>
More information about the gpfsug-discuss
mailing list