<div dir="ltr">John,<div><br></div><div>I think a "philosophical" difference between GPFS code and newer filesystems which were written later, in the age of "commodity hardware", is that GPFS expects the underlying hardware to be very reliable. So "disks" are typically RAID arrays available via multiple paths. And network links should have no errors, and be highly reliable, etc. GPFS does not detect these things well as it does not expect them to fail.</div><div><br></div><div>That's why you see some discussions around "improving network diagnostics" and "improving troubleshooting tools" and things like that. </div><div><br></div><div>Having a failed NSD is highly unusual for a GPFS system and you should design your system so that situation does not happen.</div><div><br></div><div>In your example here, if data is striped across two NSDs and one of them becomes inaccessible, when a client tries to write, it should get an I/O error, and perhaps even unmount the filesystem (depending on where you metadata lives).</div><div><br></div><div>Regards,</div><div>Alex</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 13, 2017 at 5:56 AM, John Hearns <span dir="ltr"><<a href="mailto:john.hearns@asml.com" target="_blank">john.hearns@asml.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_-1681735852914078392WordSection1">
<p class="MsoNormal">I have set up a small testbed, consisting of three nodes. Two of the nodes have a disk which is being used as an NSD.<u></u><u></u></p>
<p class="MsoNormal">This is being done for some preparation for fun and games with some whizzy new servers. The testbed has spinning drives.<u></u><u></u></p>
<p class="MsoNormal">I have created two NSDs and have set the data replication to 1 (this is deliberate).<u></u><u></u></p>
<p class="MsoNormal">I am trying to fail an NSD and find which files have parts on the failed NSD.<u></u><u></u></p>
<p class="MsoNormal">A first test with ‘mmdeldisk’ didn’t have much effect as SpectrumScale is smart enough to copy the data off the drive.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I now take the drive offline and delete it by <u></u><u></u></p>
<p class="MsoNormal">echo offline > /sys/block/sda/device/state<u></u><u></u></p>
<p class="MsoNormal">echo 1 > /sys/block/sda/delete<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Short of going to the data centre and physically pulling the drive that’s a pretty final way of stopping access to a drive.<u></u><u></u></p>
<p class="MsoNormal">I then wrote 100 files to the filesystem, the node with the NSD did log “rejecting I/O to offline device”<u></u><u></u></p>
<p class="MsoNormal">However mmlsdisk <filesystem> says that this disk is status ‘ready’<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am going to stop that NSD and run an mmdeldisk – at which point I do expect things to go south rapidly.<u></u><u></u></p>
<p class="MsoNormal">I just am not understanding at what point a failed write would be detected? Or once a write fails are all the subsequent writes
<u></u><u></u></p>
<p class="MsoNormal">Routed off to the active NSD(s) ??<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Sorry if I am asking an idiot question.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><a href="mailto:Inspector.clouseau@surete.fr" target="_blank">Inspector.clouseau@surete.fr</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
-- The information contained in this communication and any attachments is confidential and may be privileged, and is for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. Unless explicitly stated
otherwise in the body of this communication or the attachment thereto (if any), the information is provided on an AS-IS basis without any express or implied warranties or liabilities. To the extent you are relying on this information, you are doing so at your
own risk. If you are not the intended recipient, please notify the sender immediately by replying to this message and destroy all copies of this message and any attachments. Neither the sender nor the company/group of companies he or she represents shall be
liable for the proper and complete transmission of the information contained in this communication, or for any delay in its receipt.
</div>
<br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br></div>