<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:12.0pt">Hi </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Lohit, Nathan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Would you be willing to share some more details about your setup? We are just getting started here and I would like to hear about what your configuration looks like. Direct email to me is fine, thanks.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Bob Oesterlin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt">Sr Principal Storage Engineer, Nuance<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:12.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black"><gpfsug-discuss-bounces@spectrumscale.org> on behalf of "valleru@cbio.mskcc.org" <valleru@cbio.mskcc.org><br>
<b>Reply-To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Date: </b>Thursday, April 26, 2018 at 9:45 AM<br>
<b>To: </b>gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Singularity + GPFS<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div name="messageBodySection">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">We do run Singularity + GPFS, on our production HPC clusters.
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Most of the time things are fine without any issues.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">However, i do see a significant performance loss when running some applications on singularity containers with GPFS.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">As of now, the applications that have severe performance issues with singularity on GPFS - seem to be because of “mmap io”. (Deep learning applications)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">When i run the same application on bare metal, they seem to have a huge difference in GPFS IO when compared to running on singularity containers.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">I am yet to raise a PMR about this with IBM.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">I have not seen performance degradation for any other kind of IO, but i am not sure.<o:p></o:p></span></p>
</div>
</div>
<div name="messageSignatureSection">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
Regards,<br>
Lohit<o:p></o:p></span></p>
</div>
<div name="messageReplySection">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
On Apr 26, 2018, 10:35 AM -0400, Nathan Harper <nathan.harper@cfms.org.uk>, wrote:<br>
<br>
<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #1ABC9C 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">We are running on a test system at the moment, and haven't run into any issues yet, but so far it's only been 'hello world' and running FIO.
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">I'm interested to hear about experience with MPI-IO within Singularity.<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">On 26 April 2018 at 15:20, Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a>> wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #E67E22 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Anyone (including IBM) doing any work in this area? I would appreciate hearing from you.<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Arial",sans-serif"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Bob Oesterlin<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Sr Principal Storage Engineer, Nuance<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"> <o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=OTGc6UhyurUT-JrFZxFuLp_VKCTGGknEbzjMSKUa-QQ&e=" target="_blank">
spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=hPDughTsSwQUdR8KHv4gxXMumoGw3jXLgGGXG1MQrJM&e=" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
<br clear="all">
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">--<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="background:white"><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Nathan</span></b><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#515058"> </span></b><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Harper</span></b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#00AEFF"> // </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#222222">IT
Systems Lead</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__cfms.org.uk_news-2Devents_events_2018_july_farnborough-2Dinternational-2Dairshow-2D2018_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=SAiH6rxM-6WYWwUBRecbBS9Yr7f4adg4GizgVJXIAqI&e=" target="_blank"><span style="color:#222222;text-decoration:none"><span style="color:blue;border:solid windowtext 1.0pt;padding:0in"><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="_x0000_i1026" src="cid:~WRD000.jpg" alt="Image removed by sender."></span></span></a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><br>
</span><strong><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">e: </span></strong><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"><a href="mailto:nathan.harper@cfms.org.uk" target="_blank"><span style="color:#515058">nathan.harper@cfms.org.uk</span></a> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">t</span></b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">:</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> 0117
906 1104 </span><b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">m</span></b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">:</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222">
0787 551 0891 </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF"> <strong><span style="font-family:"Arial",sans-serif">w: </span></strong></span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cfms.org.uk_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=KAbz_Md-BsCIrN6LDOb6Yc7sriwSq5IeO4aaxRGQfjc&e=" target="_blank"><span style="color:#515058">www.cfms.org.uk</span></a> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">CFMS Services Ltd</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF"> // </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Bristol
& Bath Science Park</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Dirac
Crescent</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Emersons
Green</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Bristol</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00AEFF">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">BS16
7FR</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222;border:solid windowtext 1.0pt;padding:0in"><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="_x0000_i1025" src="cid:~WRD000.jpg" alt="Image removed by sender."></span><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:#515058">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222"> <br>
</span><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:#515058">CFMS Services Ltd registered office // 43 Queens Square // Bristol // BS1 4QP</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<o:p></o:p></span></p>
</blockquote>
</div>
</div>
</body>
</html>