<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >I agree, this sounds like hardware. If you haven't already, checking 'dmesg' or /var/log/messages (or syslog) on the nsd19jbod1 server may highlight any disk/adapter issues.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Also, as Olaf mentioned, you may want to engage support via a PMR.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Good Luck,</div>
<div dir="ltr" >Nikhil</div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Olaf Weiser" <olaf.weiser@de.ibm.com><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss@spectrumscale.org<br>Cc: gpfsug-discuss@spectrumscale.org<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Disk in unrecovered state<br>Date: Tue, Jan 12, 2021 8:30 AM<br>
<div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" >Hallo Iban,</div>
<div dir="ltr" > </div>
<div dir="ltr" >this seems to be a hardware issue</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Input/output error</font></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><font size="2" face="Default Monospace,Courier New,Courier,monospace" >just try /make sure.. that you really can read from the disk .. all NSDs from all of its NSD servers</font></div>
<div dir="ltr" ><font size="2" ><font face="Default Monospace,Courier New,Courier,monospace" >so to say.. its most important, that the NSD is accessible on the primary NSD server.. as long this primary NSD server is up n running</font></font></div>
<div dir="ltr" > </div>
<div dir="ltr" ><font size="2" face="Default Monospace,Courier New,Courier,monospace" >either use dd (be carfully ;-) .. only try to READ ) </font></div>
<div dir="ltr" >e.g. dd if=/dev/$yourblockdevice bs=4k count=2 | od -xc</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" ><font size="2" >or </font></div>
<div dir="ltr" ><font size="2" >mmfsadm test ...(if you are familiar with that<font face="Default Monospace,Courier New,Courier,monospace" > - if not.. feel free, open a problem record.. and we'll guide your through ..)</font></font></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<div dir="ltr" >cheers</div>
<div dir="ltr" >olaf</div>
<div dir="ltr" > </div>
<div dir="ltr" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" style="font-family:Arial, Helvetica, sans-serif;font-size:10pt" ><div dir="ltr" > </div></div></div></div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" data-history-expanded="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: Iban Cabrillo <cabrillo@ifca.unican.es><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug-discuss <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: [gpfsug-discuss] Disk in unrecovered state<br>Date: Tue, Jan 12, 2021 4:59 PM<br>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >Hi Renar,<br> The version we are installed is 5.0.4-3, and the paths to these wrong disks seems to be fine:<br> <br> [root@gpfs06 ~]# mmlsnsd -m| grep nsd18jbod1<br> nsd18jbod1 0A0A00675EE76CF5 /dev/sds gpfs05.ifca.es server node<br> nsd18jbod1 0A0A00675EE76CF5 /dev/sdby gpfs06.ifca.es server node<br>[root@gpfs06 ~]# mmlsnsd -m| grep nsd19jbod1<br> nsd19jbod1 0A0A00665EE76CF6 /dev/sdt gpfs05.ifca.es server node<br> nsd19jbod1 0A0A00665EE76CF6 /dev/sdaa gpfs06.ifca.es server node<br>[root@gpfs06 ~]# mmlsnsd -m| grep nsd19jbod2<br> nsd19jbod2 0A0A00695EE79A12 /dev/sdt gpfs07.ifca.es server node<br> nsd19jbod2 0A0A00695EE79A12 /dev/sdat gpfs08.ifca.es server node<br>[root@gpfs06 ~]# mmlsnsd -m| grep nsd24jbod2<br> nsd24jbod2 0A0A00685EE79749 /dev/sdbn gpfs07.ifca.es server node<br> nsd24jbod2 0A0A00685EE79749 /dev/sdcg gpfs08.ifca.es server node<br>[root@gpfs06 ~]# mmlsnsd -m| grep nsd57jbod1<br> nsd57jbod1 0A0A00665F243CE1 /dev/sdbg gpfs05.ifca.es server node<br> nsd57jbod1 0A0A00665F243CE1 /dev/sdbx gpfs06.ifca.es server node<br>[root@gpfs06 ~]# mmlsnsd -m| grep nsd61jbod1<br> nsd61jbod1 0A0A00665F243CFA /dev/sdbk gpfs05.ifca.es server node<br> nsd61jbod1 0A0A00665F243CFA /dev/sdy gpfs06.ifca.es server node<br>[root@gpfs06 ~]# mmlsnsd -m| grep nsd71jbod1<br> nsd71jbod1 0A0A00665F243D38 /dev/sdbu gpfs05.ifca.es server node<br> nsd71jbod1 0A0A00665F243D38 /dev/sdbv gpfs06.ifca.es server node<br><br>trying to start 19jbod1 again:<br>[root@gpfs06 ~]# mmchdisk gpfs2 start -d nsd19jbod1<br>mmnsddiscover: Attempting to rediscover the disks. This may take a while ...<br>mmnsddiscover: Finished.<br>gpfs06.ifca.es: Rediscovered nsd server access to nsd19jbod1.<br>gpfs05.ifca.es: Rediscovered nsd server access to nsd19jbod1.<br>Failed to open gpfs2.<br>Log recovery failed.<br>Input/output error<br>Initial disk state was updated successfully, but another error may have changed the state again.<br>mmchdisk: Command failed. Examine previous error messages to determine cause.<br><br>Regards, I<br><br>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font><br> </div></blockquote>
<div dir="ltr" > </div></div>
<div><font size="2" face="Default Monospace,Courier New,Courier,monospace" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> </font></div></blockquote>
<div dir="ltr" > </div></div><BR>