<div dir="ltr">We had an issue with a particular application writing out output in parallel - (I think) including gpfs.h seemed to fix the problem, but we might also have had a clockskew issue on the compute nodes at the same time, so we aren't sure exactly which fixed it.<div><br></div><div>My chaos monkeys aren't those that resist guidance, but instead are the ones that will employ all the tools at their disposal to improve performance. A lot of our applications aren't doing MPI-IO, so my very capable parallel filesystem is idling while a single rank is reading/writing. However, some will hit the filesystem much harder or exercise less used functionality, and I'm keen to make sure that works through Singularity as well.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 26 April 2018 at 16:31, David Johnson <span dir="ltr"><<a href="mailto:david_johnson@brown.edu" target="_blank">david_johnson@brown.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Regarding MPI-IO, how do you mean “building the applications against GPFS”? <div>We try to advise our users about things to avoid, but we have some poster-ready </div><div>“chaos monkeys” as well, who resist guidance. What apps do your users favor?</div><div>Molpro is one of our heaviest apps right now.</div><div>Thanks,</div><div> — ddj<div><div class="h5"><br><div><br><blockquote type="cite"><div>On Apr 26, 2018, at 11:25 AM, Nathan Harper <<a href="mailto:nathan.harper@cfms.org.uk" target="_blank">nathan.harper@cfms.org.uk</a>> wrote:</div><br class="m_6902501731328874748Apple-interchange-newline"><div><div dir="ltr">Happy to share on the list in case anyone else finds it useful:<div><br></div><div>We use GPFS for home/scratch on our HPC clusters, supporting engineering applications, so 95+% of our jobs are multi-node MPI. We have had some questions/concerns about GPFS+Singularity+MPI-IO, as we've had issues with GPFS+MPI-IO in the past that was solved by building the applications against GPFS. If users start using Singularity containers, we then can't guarantee how the contained applications have been built.</div><div><br></div><div>I've got a small test system (2 nsd nodes, 6 compute nodes) to see if we can break it, before we deploy onto our production systems. Everything seems to be ok under synthetic benchmarks, but I've handed over to one of my chaos monkey users to let him do his worst. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On 26 April 2018 at 15:53, Yugendra Guvvala <span dir="ltr"><<a href="mailto:yguvvala@cambridgecomputer.com" target="_blank">yguvvala@cambridgecomputer.<wbr>com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">I am interested to learn this too. So please add me sending a direct mail. <br><br><div id="m_6902501731328874748m_-3404058168981204889AppleMailSignature">Thanks,<div>Yugi</div></div><div><div class="m_6902501731328874748h5"><div><br>On Apr 26, 2018, at 10:51 AM, Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a>> wrote:<br><br></div><blockquote type="cite"><div>
<div class="m_6902501731328874748m_-3404058168981204889WordSection1"><p class="MsoNormal"><span style="font-size:12.0pt">Hi </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Lohit, Nathan<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt">Would you be willing to share some more details about your setup? We are just getting started here and I would like to hear about what your configuration looks like. Direct email to me is fine, thanks.<u></u><u></u></span></p>
<div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-size:12.0pt">Bob Oesterlin<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt">Sr Principal Storage Engineer, Nuance<u></u><u></u></span></p>
</div><p class="MsoNormal"><span style="font-size:12.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:12pt">From: </span></b><span style="font-size:12pt"><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank">gpfsug-discuss-bounces@spectr<wbr>umscale.org</a>> on behalf of "<a href="mailto:valleru@cbio.mskcc.org" target="_blank">valleru@cbio.mskcc.org</a>" <<a href="mailto:valleru@cbio.mskcc.org" target="_blank">valleru@cbio.mskcc.org</a>><br>
<b>Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
<b>Date: </b>Thursday, April 26, 2018 at 9:45 AM<br>
<b>To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank">gpfsug-discuss@spectrumscale.<wbr>org</a>><br>
<b>Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Singularity + GPFS<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><u></u> <u></u></p>
</div>
<div name="messageBodySection"><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">We do run Singularity + GPFS, on our production HPC clusters.
<u></u><u></u></span></p>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Most of the time things are fine without any issues.<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">However, i do see a significant performance loss when running some applications on singularity containers with GPFS.<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">As of now, the applications that have severe performance issues with singularity on GPFS - seem to be because of “mmap io”. (Deep learning applications)<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">When i run the same application on bare metal, they seem to have a huge difference in GPFS IO when compared to running on singularity containers.<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">I am yet to raise a PMR about this with IBM.<u></u><u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">I have not seen performance degradation for any other kind of IO, but i am not sure.<u></u><u></u></span></p>
</div>
</div>
<div name="messageSignatureSection"><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
Regards,<br>
Lohit<u></u><u></u></span></p>
</div>
<div name="messageReplySection"><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
On Apr 26, 2018, 10:35 AM -0400, Nathan Harper <<a href="mailto:nathan.harper@cfms.org.uk" target="_blank">nathan.harper@cfms.org.uk</a>>, wrote:<br>
<br>
<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #1abc9c 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt">
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">We are running on a test system at the moment, and haven't run into any issues yet, but so far it's only been 'hello world' and running FIO.
<u></u><u></u></span></p>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">I'm interested to hear about experience with MPI-IO within Singularity.<u></u><u></u></span></p>
</div>
</div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">On 26 April 2018 at 15:20, Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank">Robert.Oesterlin@nuance.com</a>> wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #e67e22 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt">
<div>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Anyone (including IBM) doing any work in this area? I would appreciate hearing from you.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Bob Oesterlin<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">Sr Principal Storage Engineer, Nuance<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"> <u></u><u></u></span></p>
</div>
</div><p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=OTGc6UhyurUT-JrFZxFuLp_VKCTGGknEbzjMSKUa-QQ&e=" target="_blank">
spectrumscale.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=hPDughTsSwQUdR8KHv4gxXMumoGw3jXLgGGXG1MQrJM&e=" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><u></u><u></u></span></p>
</blockquote>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><br>
<br clear="all">
<u></u><u></u></span></p>
<div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif"><u></u> <u></u></span></p>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">--<u></u><u></u></span></p>
<div>
<div><p class="MsoNormal" style="background:white"><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Nathan</span></b><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#515058"> </span></b><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222">Harper</span></b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#00aeff"> // </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#222222">IT
Systems Lead</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><u></u><u></u></span></p>
</div>
<div><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><u></u> <u></u></span></p>
</div>
<div><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__cfms.org.uk_news-2Devents_events_2018_july_farnborough-2Dinternational-2Dairshow-2D2018_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=SAiH6rxM-6WYWwUBRecbBS9Yr7f4adg4GizgVJXIAqI&e=" target="_blank"><span style="color:#222222;text-decoration:none"><span style="color:blue;border:solid windowtext 1.0pt;padding:0in"><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="m_6902501731328874748m_-3404058168981204889_x0000_i1026" alt="Image removed by sender."></span></span></a><u></u><u></u></span></p>
</div>
<div><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><br>
</span><strong><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">e: </span></strong><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"><a href="mailto:nathan.harper@cfms.org.uk" target="_blank"><span style="color:#515058">nathan.harper@cfms.org.uk</span></a> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"><wbr> </span><b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">t</span></b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">:</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> 0117
906 1104 </span><b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">m</span></b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">:</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222">
0787 551 0891 </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff"> <strong><span style="font-family:"Arial",sans-serif">w: </span></strong></span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cfms.org.uk_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=KAbz_Md-BsCIrN6LDOb6Yc7sriwSq5IeO4aaxRGQfjc&e=" target="_blank"><span style="color:#515058">www.cfms.org.uk</span></a> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><u></u><u></u></span></p>
</div>
<div><p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">CFMS Services Ltd</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff"> // </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Bristol
& Bath Science Park</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Dirac
Crescent</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Emersons
Green</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">Bristol</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058">BS16
7FR</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><u></u><u></u></span></p>
</div>
<div><p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222;border:solid windowtext 1.0pt;padding:0in"><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="m_6902501731328874748m_-3404058168981204889_x0000_i1025" alt="Image removed by sender."></span><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222"> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222"><u></u><u></u></span></p>
</div>
<div><p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:#515058">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222"> <br>
</span><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:#515058">CFMS Services Ltd registered office // 43 Queens Square // Bristol // BS1 4QP</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222"><u></u><u></u></span></p>
</div>
</div>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif">______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><u></u><u></u></span></p>
</blockquote>
</div>
</div>
</div></blockquote><blockquote type="cite"><div><span>______________________________<wbr>_________________</span><br><span>gpfsug-discuss mailing list</span><br><span>gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank">spectrumscale.org</a></span><br><span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a></span><br></div></blockquote></div></div></div><br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org/" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/list<wbr>info/gpfsug-discuss</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_6902501731328874748gmail_signature" data-smartmail="gmail_signature"><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><b style="font-family:Arial;font-size:small">Nathan<span style="color:rgb(81,80,88)"> </span>Harper</b><span style="font-family:Arial;font-size:small;color:rgb(0,174,255)"> // </span><font size="2" style="font-family:Arial">IT Systems Lead</font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><font size="2" style="font-family:Arial"><br></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><a href="https://cfms.org.uk/news-events/events/2018/july/farnborough-international-airshow-2018/" target="_blank"><img style="width:500px;height:105px" src="https://cfms.org.uk/media/267028/fia-email-sig-blue-dark-3.jpg"></a></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><br><font face="Arial" size="1"><strong><span style="color:rgb(0,174,255)">e: </span></strong><a href="mailto:nathan.harper@cfms.org.uk" style="color:rgb(81,80,88)" target="_blank">nathan.harper@cfms.org.uk</a> <span style="color:rgb(0,174,255)"> </span><font color="#222222"><wbr> </font><font color="#00aeff"><b>t</b>:</font><font color="#222222"> 0117 906 1104</font> <font color="#222222"> </font><font color="#00aeff"><b>m</b>:</font><font color="#222222"> 0787 551 0891 </font><span style="color:rgb(0,174,255)"> <strong>w: </strong></span><a href="http://www.cfms.org.uk/" rel="noreferrer" style="color:rgb(81,80,88)" target="_blank">www.cfms.org.uk</a> <span style="color:rgb(0,174,255)"> </span></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><font face="Arial" size="1"><span style="color:rgb(81,80,88)">CFMS Services Ltd</span><span style="color:rgb(0,174,255)"> // </span><span style="color:rgb(81,80,88)">Bristol & Bath Science Park</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">Dirac Crescent</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">Emersons Green</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">Bristol</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">BS16 7FR</span> </font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><font face="Verdana" style="font-size:small"><img src="http://cfms.org.uk/images/logo.png"> </font></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;background-color:rgb(255,255,255)"><span style="font-family:Verdana"><span style="color:rgb(81,80,88);font-size:xx-small">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span> <br><span style="color:rgb(81,80,88);font-size:xx-small">CFMS Services Ltd registered office // </span></span><font color="#515058" face="Verdana"><span style="font-size:xx-small">43 Queens Square // Bristol // BS1 4QP</span></font></div></div>
</div>
______________________________<wbr>_________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br></div></blockquote></div><br></div></div></div></div><br>______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><b style="font-family:Arial;font-size:small">Nathan<span style="color:rgb(81,80,88)"> </span>Harper</b><span style="font-family:Arial;font-size:small;color:rgb(0,174,255)"> // </span><font size="2" style="font-family:Arial">IT Systems Lead</font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><font size="2" style="font-family:Arial"><br></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><a href="https://cfms.org.uk/news-events/events/2018/july/farnborough-international-airshow-2018/" target="_blank"><img style="width:500px;height:105px" src="https://cfms.org.uk/media/267028/fia-email-sig-blue-dark-3.jpg"></a></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><br><font face="Arial" size="1"><strong><span style="color:rgb(0,174,255)">e: </span></strong><a href="mailto:nathan.harper@cfms.org.uk" style="color:rgb(81,80,88)" target="_blank">nathan.harper@cfms.org.uk</a> <span style="color:rgb(0,174,255)"> </span><font color="#222222"> </font><font color="#00aeff"><b>t</b>:</font><font color="#222222"> 0117 906 1104</font> <font color="#222222"> </font><font color="#00aeff"><b>m</b>:</font><font color="#222222"> 0787 551 0891 </font><span style="color:rgb(0,174,255)"> <strong>w: </strong></span><a href="http://www.cfms.org.uk/" rel="noreferrer" style="color:rgb(81,80,88)" target="_blank">www.cfms.org.uk</a> <span style="color:rgb(0,174,255)"> </span></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><font face="Arial" size="1"><span style="color:rgb(81,80,88)">CFMS Services Ltd</span><span style="color:rgb(0,174,255)"> // </span><span style="color:rgb(81,80,88)">Bristol & Bath Science Park</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">Dirac Crescent</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">Emersons Green</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">Bristol</span> <span style="color:rgb(0,174,255)">//</span> <span style="color:rgb(81,80,88)">BS16 7FR</span> </font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)"><font face="Verdana" style="font-size:small"><img src="http://cfms.org.uk/images/logo.png"> </font></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;background-color:rgb(255,255,255)"><span style="font-family:Verdana"><span style="color:rgb(81,80,88);font-size:xx-small">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span> <br><span style="color:rgb(81,80,88);font-size:xx-small">CFMS Services Ltd registered office // </span></span><font color="#515058" face="Verdana"><span style="font-size:xx-small">43 Queens Square // Bristol // BS1 4QP</span></font></div></div>
</div>