<div dir="ltr">Hello,<div><br></div><div>there are multiple reasons why the descriptors can not be found .</div><div><br></div><div>there was a recent change in firmware behaviors on multiple servers that restore the GPT table from a disk if the disk was used as a OS disk before used as GPFS disks. some infos here : <a href="https://www.ibm.com/developerworks/community/forums/html/topic?id=27f98aab-aa41-41f4-b6b7-c87d3ce87b9e" target="_blank">https://www.ibm.com/developerworks/community/forums/html/topic?id=27f98aab-aa41-41f4-b6b7-c87d3ce87b9e</a></div><div><br></div><div>if thats the case there is a procedure to restore them.</div><div><br></div><div>it could also be something very trivial , e.g. that your multipath mapping changed and your nsddevice file actually just prints out devices instead of scanning them and create a list on the fly , so GPFS ignores the new path to the disks. </div><div>in any case , opening a PMR and work with Support is the best thing to do before causing any more damage. </div><div>if the file-system is still mounted don't unmount it under any circumstances as Support needs to extract NSD descriptor information from it to restore them easily. </div><div><br></div><div>Sven</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 29, 2014 at 8:31 AM, Jared David Baker <span dir="ltr"><<a href="mailto:Jared.Baker@uwyo.edu" target="_blank">Jared.Baker@uwyo.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-family:Consolas">Hello all,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">I’m hoping that somebody can shed some light on a problem that I experienced yesterday. I’ve been working with GPFS for a couple months as an admin now, but I’ve come across a problem that I’m unable to
see the answer to. Hopefully the solution is not listed somewhere blatantly on the web, but I spent a fair amount of time looking last night. Here is the situation: yesterday, I needed to update some firmware on a Mellanox HCA FDR14 card and reboot one of
our GPFS servers and repeat for the sister node (IBM x3550 and DCS3850) as HPSS for our main campus cluster. However, upon reboot, the server seemed to lose the path mappings to the multipath devices for the NSDs. Output below:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]# mmlsnsd -m -f gscratch<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Disk name NSD volume ID Device Node name Remarks
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">---------------------------------------------------------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun0 0A62001B54235577 - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun0 0A62001B54235577 - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun10 0A62001C542355AA - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun10 0A62001C542355AA - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun2 0A62001C54235581 - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun2 0A62001C54235581 - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun4 0A62001B5423558B - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun4 0A62001B5423558B - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun6 0A62001C54235595 - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun6 0A62001C54235595 - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun8 0A62001B5423559F - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun8 0A62001B5423559F - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun1 0A62001B5423557C - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun1 0A62001B5423557C - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun11 0A62001C542355AF - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun11 0A62001C542355AF - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun3 0A62001C54235586 - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun3 0A62001C54235586 - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun5 0A62001B54235590 - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun5 0A62001B54235590 - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun7 0A62001C5423559A - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun7 0A62001C5423559A - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun9 0A62001B542355A4 - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun9 0A62001B542355A4 - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]#<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Also, the system was working fantastically before the reboot, but now I’m unable to mount the GPFS filesystem. The disk names look like they are there and mapped to the NSD volume ID, but there is no Device.
I’ve created the /var/mmfs/etc/nsddevices script and it has the following output with user return 0:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]# /var/mmfs/etc/nsddevices
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31a_lun0 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31a_lun10 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31a_lun2 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31a_lun4 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31a_lun6 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31a_lun8 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31b_lun1 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31b_lun11 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31b_lun3 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31b_lun5 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31b_lun7 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mapper/dcs3800u31b_lun9 dmm<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]#<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">That output looks correct to me based on the documentation. So I went digging in the GPFS log file and found this relevant information:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.405 2014: I/O to NSD disk, dcs3800u31a_lun0, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.481 2014: I/O to NSD disk, dcs3800u31b_lun1, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.555 2014: I/O to NSD disk, dcs3800u31a_lun2, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.629 2014: I/O to NSD disk, dcs3800u31b_lun3, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.703 2014: I/O to NSD disk, dcs3800u31a_lun4, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.775 2014: I/O to NSD disk, dcs3800u31b_lun5, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.844 2014: I/O to NSD disk, dcs3800u31a_lun6, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.919 2014: I/O to NSD disk, dcs3800u31b_lun7, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:48.989 2014: I/O to NSD disk, dcs3800u31a_lun8, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:49.060 2014: I/O to NSD disk, dcs3800u31b_lun9, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:49.128 2014: I/O to NSD disk, dcs3800u31a_lun10, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Tue Oct 28 23:44:49.199 2014: I/O to NSD disk, dcs3800u31b_lun11, fails. No such NSD locally found.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Okay, so the NSDs don’t seem to be able to be found, so I attempt to rediscover the NSD by executing the command mmnsddiscover:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]# mmnsddiscover
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mmnsddiscover: Attempting to rediscover the disks. This may take a while ...
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">mmnsddiscover: Finished.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]#<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">I was hoping that finished, but then upon restarting GPFS, there was no success. Verifying with mmlsnsd -X -f gscratch<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]# mmlsnsd -X -f gscratch<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Disk name NSD volume ID Device Devtype Node name Remarks
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">---------------------------------------------------------------------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun0 0A62001B54235577 - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun0 0A62001B54235577 - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun10 0A62001C542355AA - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun10 0A62001C542355AA - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun2 0A62001C54235581 - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun2 0A62001C54235581 - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun4 0A62001B5423558B - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun4 0A62001B5423558B - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun6 0A62001C54235595 - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun6 0A62001C54235595 - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun8 0A62001B5423559F - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31a_lun8 0A62001B5423559F - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun1 0A62001B5423557C - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun1 0A62001B5423557C - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun11 0A62001C542355AF - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun11 0A62001C542355AF - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun3 0A62001C54235586 - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun3 0A62001C54235586 - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun5 0A62001B54235590 - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun5 0A62001B54235590 - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun7 0A62001C5423559A - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun7 0A62001C5423559A - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun9 0A62001B542355A4 - - mminsd5.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">dcs3800u31b_lun9 0A62001B542355A4 - - mminsd6.infini (not found) server node<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">[root@mmmnsd5 ~]#<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">I’m wondering if somebody has seen this type of issue before? Will recreating my NSDs destroy the filesystem? I’m thinking that all the data is intact, but there is no crucial data on this file system
yet, so I could recreate the file system, but I would like to learn how to solve a problem like this. Thanks for all help and information.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Regards,<span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></span></p><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas">Jared <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:Consolas"><u></u> <u></u></span></p>
</font></span></div>
</div>
<br>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://gpfsug.org" target="_blank">gpfsug.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>