<div dir="auto">It may depend on which state the NSDs are in with respect to the node in question. If from that node you use 'mmfsadm dump nsd | egrep "moved|error|broken" ' and see anything, that might be it.<div dir="auto"><br></div><div dir="auto">One or two of those states can be fixed by mmnsddiscover, the other(s) require a kick of mmfsd to get the NSDs back. I never remember which is which.</div><div dir="auto"><br></div><div dir="auto">-Jordan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 25, 2019, 13:13 Jan-Frode Myklebust <<a href="mailto:janfrode@tanso.net" target="_blank" rel="noreferrer">janfrode@tanso.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="auto">I’ve had a situation recently where mmnsddiscover didn’t help, but mmshutdown/mmstartup on that node did fix it.</div></div><div dir="auto"><br></div><div dir="auto">This was with v5.0.2-3 on ppc64le.</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"> -jf</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">tir. 25. jun. 2019 kl. 17:02 skrev Son Truong <<a href="mailto:son.truong@bristol.ac.uk" rel="noreferrer noreferrer" target="_blank">son.truong@bristol.ac.uk</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hello Renar,<br>
<br>
Thanks for that command, very useful and I can now see the problematic NSDs are all served remotely.<br>
<br>
I have double checked the multipath and devices and I can see these NSDs are available locally.<br>
<br>
How do I get GPFS to recognise this and server them out via 'localhost'?<br>
<br>
mmnsddiscover -d <NSD> seemed to have brought two of the four problematic NSDs back to being served locally, but the other two are not behaving. I have double checked the availability of these devices and their multipaths but everything on that side seems fine.<br>
<br>
Any more ideas?<br>
<br>
Regards,<br>
Son<br>
<br>
<br>
---------------------------<br>
<br>
Message: 2<br>
Date: Tue, 25 Jun 2019 12:10:53 +0000<br>
From: "Grunenberg, Renar" <<a href="mailto:Renar.Grunenberg@huk-coburg.de" rel="noreferrer noreferrer" target="_blank">Renar.Grunenberg@huk-coburg.de</a>><br>
To: "<a href="mailto:gpfsug-discuss@spectrumscale.org" rel="noreferrer noreferrer" target="_blank">gpfsug-discuss@spectrumscale.org</a>"<br>
<<a href="mailto:gpfsug-discuss@spectrumscale.org" rel="noreferrer noreferrer" target="_blank">gpfsug-discuss@spectrumscale.org</a>><br>
Subject: Re: [gpfsug-discuss] rescan-scsi-bus.sh and "Local access to<br>
NSD failed with EIO, switching to access the disk remotely."<br>
Message-ID: <<a href="mailto:e0f3c96e05cf4234abe5d4779df4ed2b@SMXRF105.msg.hukrf.de" rel="noreferrer noreferrer" target="_blank">e0f3c96e05cf4234abe5d4779df4ed2b@SMXRF105.msg.hukrf.de</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hallo Son,<br>
<br>
you can check the access to the nsd with mmlsdisk <fsname> -m. This give you a colum like ?IO performed on node?. On NSD-Server you should see localhost, on nsd-client you see the hostig nsd-server per device.<br>
<br>
Regards Renar<br>
<br>
<br>
Renar Grunenberg<br>
Abteilung Informatik - Betrieb<br>
<br>
HUK-COBURG<br>
Bahnhofsplatz<br>
96444 Coburg<br>
Telefon: 09561 96-44110<br>
Telefax: 09561 96-44104<br>
E-Mail: <a href="mailto:Renar.Grunenberg@huk-coburg.de" rel="noreferrer noreferrer" target="_blank">Renar.Grunenberg@huk-coburg.de</a><br>
Internet: <a href="http://www.huk.de" rel="noreferrer noreferrer noreferrer" target="_blank">www.huk.de</a><br>
________________________________<br>
HUK-COBURG Haftpflicht-Unterst?tzungs-Kasse kraftfahrender Beamter Deutschlands a. G. in Coburg Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021 Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.<br>
Vorstand: Klaus-J?rgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav Her?y, Dr. J?rg Rheinl?nder (stv.), Sarah R?ssler, Daniel Thomas.<br>
________________________________<br>
Diese Nachricht enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen.<br>
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.<br>
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist nicht gestattet.<br>
<br>
This information may contain confidential and/or privileged information.<br>
If you are not the intended recipient (or have received this information in error) please notify the sender immediately and destroy this information.<br>
Any unauthorized copying, disclosure or distribution of the material in this information is strictly forbidden.<br>
________________________________<br>
Von: <a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" rel="noreferrer noreferrer" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a> <<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" rel="noreferrer noreferrer" target="_blank">gpfsug-discuss-bounces@spectrumscale.org</a>> Im Auftrag von Son Truong<br>
Gesendet: Dienstag, 25. Juni 2019 13:38<br>
An: <a href="mailto:gpfsug-discuss@spectrumscale.org" rel="noreferrer noreferrer" target="_blank">gpfsug-discuss@spectrumscale.org</a><br>
Betreff: [gpfsug-discuss] rescan-scsi-bus.sh and "Local access to NSD failed with EIO, switching to access the disk remotely."<br>
<br>
Hello,<br>
<br>
I wonder if anyone has seen this? I am (not) having fun with the rescan-scsi-bus.sh command especially with the -r switch. Even though there are no devices removed the script seems to interrupt currently working NSDs and these messages appear in the mmfs.logs:<br>
<br>
2019-06-25_06:30:48.706+0100: [I] Connected to <IP> <node> <c0n0><br>
2019-06-25_06:30:48.764+0100: [E] Local access to <NSD> failed with EIO, switching to access the disk remotely.<br>
2019-06-25_06:30:51.187+0100: [E] Local access to <NSD> failed with EIO, switching to access the disk remotely.<br>
2019-06-25_06:30:51.188+0100: [E] Local access to <NSD> failed with EIO, switching to access the disk remotely.<br>
2019-06-25_06:30:51.188+0100: [N] Connecting to <IP> <node> <c0n5><br>
2019-06-25_06:30:51.195+0100: [I] Connected to <IP> <node> <c0n5><br>
2019-06-25_06:30:59.857+0100: [N] Connecting to <IP> <node> <c0n4><br>
2019-06-25_06:30:59.863+0100: [I] Connected to <IP> <node> <c0n4><br>
2019-06-25_06:33:30.134+0100: [E] Local access to <NSD> failed with EIO, switching to access the disk remotely.<br>
2019-06-25_06:33:30.151+0100: [E] Local access to <NSD> failed with EIO, switching to access the disk remotely.<br>
<br>
These messages appear roughly at the same time each day and I?ve checked the NSDs via mmlsnsd and mmlsdisk commands and they are all ?ready? and ?up?. The multipaths to these NSDs are all fine too.<br>
<br>
Is there a way of finding out what ?access? (local or remote) a particular node has to an NSD? And is there a command to force it to switch to local access ? ?mmnsdrediscover? returns nothing and run really fast (contrary to the statement ?This may take a while? when it runs)?<br>
<br>
Any ideas appreciated!<br>
<br>
Regards,<br>
Son<br>
<br>
Son V Truong - Senior Storage Administrator Advanced Computing Research Centre IT Services, University of Bristol<br>
Email: <a href="mailto:son.truong@bristol.ac.uk" rel="noreferrer noreferrer" target="_blank">son.truong@bristol.ac.uk</a><mailto:<a href="mailto:s.truong@bristol.ac.uk" rel="noreferrer noreferrer" target="_blank">s.truong@bristol.ac.uk</a>><br>
Tel: Mobile: +44 (0) 7732 257 232<br>
Address: <a href="https://www.google.com/maps/search/31+Great+George+Street,+Bristol,+BS1+5QD?entry=gmail&source=g" rel="noreferrer noreferrer" target="_blank">31 Great George Street, Bristol, BS1 5QD</a><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20190625/db704f88/attachment.html" rel="noreferrer noreferrer noreferrer" target="_blank">http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20190625/db704f88/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer noreferrer noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
<br>
<br>
End of gpfsug-discuss Digest, Vol 89, Issue 26<br>
**********************************************<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer noreferrer noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div></div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer noreferrer noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer noreferrer noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>