<div dir="ltr">i am not sure i understand your comment with 'persistent' do you mean when you create a nsddevice on a nvme device it won't get recognized after a restart ?<div>if thats what you mean there are 2 answers , short term you need to add a /var/mmfs/etc/nsddevices script to your node that simply adds an echo for the nvme device like :</div><div><br></div><div>echo nvme0n1 generic</div><div><br></div><div>this will tell the daemon to include that device on top of all other discovered devices that we include by default (like dm-* , sd*, etc)</div><div>the longer term answer is that we have a tracking item to ad nvme* to the automatically discovered devices. </div><div><br></div><div>on your second question, given that GPFS does workload balancing across devices you don't want to add extra complexity and path length to anything , so stick with raw devices .</div><div><br></div><div>sven</div><div><br></div><div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 6, 2016 at 8:40 AM Matt Weil <<a href="mailto:mweil@wustl.edu">mweil@wustl.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
<p class="gmail_msg">Hello all,<br class="gmail_msg">
</p>
<p class="gmail_msg">Thanks for sharing that. I am setting this up on our CES nodes. In this example the nvme devices are not persistent. RHEL's default udev rules put them in /dev/disk/by-id/ persistently by serial number so I modified mmdevdiscover to look for them there.
What are others doing? custom udev rules for the nvme devices?<br class="gmail_msg">
</p>
<p class="gmail_msg">Also I have used LVM in the past to stitch multiple nvme together for better performance. I am wondering in the use case with GPFS that it may hurt performance by hindering the ability of GPFS to do direct IO or directly accessing memory. Any opinions
there?</p>
<p class="gmail_msg">Thanks</p>
<p class="gmail_msg">Matt<br class="gmail_msg">
</p></div><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg">
<div class="m_-4545806242424547591moz-cite-prefix gmail_msg">On 12/5/16 10:33 AM, Ulf Troppens wrote:<br class="gmail_msg">
</div>
<blockquote type="cite" class="gmail_msg">
<p class="gmail_msg">FYI ... in case not seen .... benchmark for LROC with NVMe<br class="gmail_msg">
<a href="http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/performance-gains-ibm-spectrum-scale.pdf" class="gmail_msg" target="_blank">http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/performance-gains-ibm-spectrum-scale.pdf</a><br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
--<br class="gmail_msg">
IBM Spectrum Scale Development - Client Engagements & Solutions Delivery<br class="gmail_msg">
Consulting IT Specialist<br class="gmail_msg">
Author "Storage Networks Explained"<br class="gmail_msg">
<br class="gmail_msg">
IBM Deutschland Research & Development GmbH<br class="gmail_msg">
Vorsitzende des Aufsichtsrats: Martina Koederitz<br class="gmail_msg">
Geschäftsführung: Dirk Wittkopp<br class="gmail_msg">
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
</p>
<fieldset class="m_-4545806242424547591mimeAttachmentHeader gmail_msg"></fieldset> <br class="gmail_msg">
<pre class="gmail_msg">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at <a href="http://spectrumscale.org" class="gmail_msg" target="_blank">spectrumscale.org</a>
<a class="m_-4545806242424547591moz-txt-link-freetext gmail_msg" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br class="gmail_msg">
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"" class="gmail_msg"><u class="gmail_msg"></u><u class="gmail_msg"></u></span>
<p class="gmail_msg"></p>
<div class="gmail_msg"></div>
<p class="MsoNormal gmail_msg"><u class="gmail_msg"></u> <u class="gmail_msg"></u></p>
</div><div bgcolor="#FFFFFF" text="#000000" class="gmail_msg"><div class="MsoNormal gmail_msg" align="center" style="text-align:center">
<hr size="2" width="100%" align="center" class="gmail_msg">
</div>
<p class="MsoNormal gmail_msg"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:gray" class="gmail_msg">The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended
recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone
or return mail.</span></p>
</div>
_______________________________________________<br class="gmail_msg">
gpfsug-discuss mailing list<br class="gmail_msg">
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" class="gmail_msg" target="_blank">spectrumscale.org</a><br class="gmail_msg">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" class="gmail_msg" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class="gmail_msg">
</blockquote></div>