<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">Two things:</div>
<div class=""><br class="">
</div>
<div class="">1) Paul’s suggestion to look at the nsddevices script was the answer I needed to fix my mmcrfs issue. Thanks.</div>
<div class=""><br class="">
</div>
<div class="">2) I am also interested in hearing if anyone is using ZFS to create the equivalent of RAID 8+2P hardware RAID 6 LUNs and presenting that to GPFS to use as disks?</div>
<div class=""><br class="">
</div>
<div class="">Thanks…</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Mar 11, 2017, at 4:37 AM, Daniel Kidger <<a href="mailto:daniel.kidger@uk.ibm.com" class="">daniel.kidger@uk.ibm.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<p class=""><font size="2" class="">On the subject of using zvols for software Raid/ replication, can ask as a quick poll, how many people are doing this?<br class="">
</font><font size="2" class=""><br class="">
</font><font size="2" class="">And any feedback on stability, tuning and performance?<br class="">
</font><font size="2" class=""><br class="">
</font><font size="2" class="">Daniel<br class="">
</font><font size="2" class="">IBM Technical Presales<br class="">
</font><font size="2" class=""><br class="">
</font><font size="2" class="">> On 10 Mar 2017, at 22:44, Aaron Knister <<a href="mailto:aaron.s.knister@nasa.gov" class="">aaron.s.knister@nasa.gov</a>> wrote:<br class="">
</font><font size="2" class="">> <br class="">
</font><font size="2" class="">> Those look like zvol's. Out of curiosity have you set sync=always on the
<br class="">
</font><font size="2" class="">> filesystem root or zvols themselves? It's my understanding that without
<br class="">
</font><font size="2" class="">> that you risk data loss since GPFS won't ever cause a sync to be issued
<br class="">
</font><font size="2" class="">> to the zvol for zfs to flush acknowledged but uncommitted writes.<br class="">
</font><font size="2" class="">> <br class="">
</font><font size="2" class="">> -Aaron<br class="">
</font><font size="2" class="">> <br class="">
</font><font size="2" class="">>> On 3/10/17 4:36 PM, Sanchez, Paul wrote:<br class="">
</font><font size="2" class="">>> See:<br class="">
</font><font size="2" class="">>> <a href="https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm" class="">
https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.0/com.ibm.spectrum.scale.v4r2.adm.doc/bl1adm_nsddevices.htm</a><br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> GPFS has a limited set of device search specs it uses to find connected<br class="">
</font><font size="2" class="">>> NSDs. When using exotic devices, you need to whitelist the devices<br class="">
</font><font size="2" class="">>> yourself using the user exit script at /var/mmfs/etc/nsddevices.<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> *From:*<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">gpfsug-discuss-bounces@spectrumscale.org</a><br class="">
</font><font size="2" class="">>> [<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" class="">mailto:gpfsug-discuss-bounces@spectrumscale.org</a>] *On Behalf Of<br class="">
</font><font size="2" class="">>> *Buterbaugh, Kevin L<br class="">
</font><font size="2" class="">>> *Sent:* Friday, March 10, 2017 3:44 PM<br class="">
</font><font size="2" class="">>> *To:* gpfsug main discussion list<br class="">
</font><font size="2" class="">>> *Subject:* [gpfsug-discuss] mmcrfs issue<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Hi All,<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> We are testing out some flash storage. I created a couple of NSDs<br class="">
</font><font size="2" class="">>> successfully (?):<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> root@nec:~/gpfs# mmlsnsd -F<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> File system Disk name NSD servers<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> ---------------------------------------------------------------------------<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> (free disk) nsd1 nec<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> (free disk) nsd2 nec<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> root@nec:~/gpfs#<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> So I tried to create a filesystem:<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> root@nec:~/gpfs# mmcrfs gpfs0 -F ~/gpfs/flash.stanza -A yes -B 1M -j<br class="">
</font><font size="2" class="">>> scatter -k all -m 1 -M 3 -Q no -r 1 -R 3 -T /gpfs0<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> GPFS: 6027-441 Unable to open disk 'nsd2' on node nec.<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> GPFS: 6027-441 Unable to open disk 'nsd1' on node nec.<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> No such device<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> No such device<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> GPFS: 6027-538 Error accessing disks.<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> mmcrfs: 6027-1200 tscrfs failed. Cannot create gpfs0<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> mmcrfs: 6027-1639 Command failed. Examine previous error messages to<br class="">
</font><font size="2" class="">>> determine cause.<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> root@nec:~/gpfs#<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Does this output from readdescraw look normal?<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> root@nec:~/gpfs# mmfsadm test readdescraw /dev/zd16<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> NSD descriptor in sector 64 of /dev/zd16<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> NSDid: 0A0023D258C1C02C format version: 1403 Label:<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Paxos sector: -1931478434 number of sectors: 8192 isPdisk: 0<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Comment: NSD descriptor for <unknown> Thu Mar 9 14:50:52 2017<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> No Disk descriptor in sector 96 of /dev/zd16<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> No FS descriptor in sector 2048 of /dev/zd16<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> root@nec:~/gpfs# mmfsadm test readdescraw /dev/zd32<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> NSD descriptor in sector 64 of /dev/zd32<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> NSDid: 0A0023D258C1C02B format version: 1403 Label:<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Paxos sector: -1880562609 number of sectors: 8192 isPdisk: 0<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Comment: NSD descriptor for <unknown> Thu Mar 9 14:50:51 2017<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> No Disk descriptor in sector 96 of /dev/zd32<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> No FS descriptor in sector 2048 of /dev/zd32<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> root@nec:~/gpfs#<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Thanks in advance, all…<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Kevin<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> —<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Kevin Buterbaugh - Senior System Administrator<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> Vanderbilt University - Advanced Computing Center for Research and Education<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">
Kevin.Buterbaugh@vanderbilt.edu</a><br class="">
</font><font size="2" class="">>> <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">mailto:Kevin.Buterbaugh@vanderbilt.edu</a>> - (615)875-9633<br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">>> _______________________________________________<br class="">
</font><font size="2" class="">>> gpfsug-discuss mailing list<br class="">
</font><font size="2" class="">>> gpfsug-discuss at <a href="http://spectrumscale.org" class="">
spectrumscale.org</a><br class="">
</font><font size="2" class="">>> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</font><font size="2" class="">>> <br class="">
</font><font size="2" class="">> <br class="">
</font><font size="2" class="">> -- <br class="">
</font><font size="2" class="">> Aaron Knister<br class="">
</font><font size="2" class="">> NASA Center for Climate Simulation (Code 606.2)<br class="">
</font><font size="2" class="">> Goddard Space Flight Center<br class="">
</font><font size="2" class="">> (301) 286-2776<br class="">
</font><font size="2" class="">> _______________________________________________<br class="">
</font><font size="2" class="">> gpfsug-discuss mailing list<br class="">
</font><font size="2" class="">> gpfsug-discuss at <a href="http://spectrumscale.org" class="">
spectrumscale.org</a><br class="">
</font><font size="2" class="">> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">
http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</font><font size="2" class="">> <br class="">
</font>Unless stated otherwise above:<br class="">
IBM United Kingdom Limited - Registered in England and Wales with number 741598. <br class="">
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU<br class="">
<br class="">
</p>
</div>
_______________________________________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>