<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Regarding MPI-IO, how do you mean “building the applications against GPFS”? <div class="">We try to advise our users about things to avoid, but we have some poster-ready </div><div class="">“chaos monkeys” as well, who resist guidance. What apps do your users favor?</div><div class="">Molpro is one of our heaviest apps right now.</div><div class="">Thanks,</div><div class=""> — ddj<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 26, 2018, at 11:25 AM, Nathan Harper <<a href="mailto:nathan.harper@cfms.org.uk" class="">nathan.harper@cfms.org.uk</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Happy to share on the list in case anyone else finds it useful:<div class=""><br class=""></div><div class="">We use GPFS for home/scratch on our HPC clusters, supporting engineering applications, so 95+% of our jobs are multi-node MPI. We have had some questions/concerns about GPFS+Singularity+MPI-IO, as we've had issues with GPFS+MPI-IO in the past that was solved by building the applications against GPFS. If users start using Singularity containers, we then can't guarantee how the contained applications have been built.</div><div class=""><br class=""></div><div class="">I've got a small test system (2 nsd nodes, 6 compute nodes) to see if we can break it, before we deploy onto our production systems. Everything seems to be ok under synthetic benchmarks, but I've handed over to one of my chaos monkey users to let him do his worst. </div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 26 April 2018 at 15:53, Yugendra Guvvala <span dir="ltr" class=""><<a href="mailto:yguvvala@cambridgecomputer.com" target="_blank" class="">yguvvala@cambridgecomputer.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto" class="">I am interested to learn this too. So please add me sending a direct mail. <br class=""><br class=""><div id="m_-3404058168981204889AppleMailSignature" class="">Thanks,<div class="">Yugi</div></div><div class=""><div class="h5"><div class=""><br class="">On Apr 26, 2018, at 10:51 AM, Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank" class="">Robert.Oesterlin@nuance.com</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class="">
<div class="m_-3404058168981204889WordSection1"><p class="MsoNormal"><span style="font-size:12.0pt" class="">Hi </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">Lohit, Nathan<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Would you be willing to share some more details about your setup? We are just getting started here and I would like to hear about what your configuration looks like. Direct email to me is fine, thanks.<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Bob Oesterlin<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class="">Sr Principal Storage Engineer, Nuance<u class=""></u><u class=""></u></span></p>
</div><p class="MsoNormal"><span style="font-size:12.0pt" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:12.0pt" class=""><u class=""></u> <u class=""></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class=""><p class="MsoNormal"><b class=""><span style="font-size: 12pt;" class="">From: </span></b><span style="font-size: 12pt;" class=""><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org" target="_blank" class="">gpfsug-discuss-bounces@<wbr class="">spectrumscale.org</a>> on behalf of "<a href="mailto:valleru@cbio.mskcc.org" target="_blank" class="">valleru@cbio.mskcc.org</a>" <<a href="mailto:valleru@cbio.mskcc.org" target="_blank" class="">valleru@cbio.mskcc.org</a>><br class="">
<b class="">Reply-To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.<wbr class="">org</a>><br class="">
<b class="">Date: </b>Thursday, April 26, 2018 at 9:45 AM<br class="">
<b class="">To: </b>gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org" target="_blank" class="">gpfsug-discuss@spectrumscale.<wbr class="">org</a>><br class="">
<b class="">Subject: </b>[EXTERNAL] Re: [gpfsug-discuss] Singularity + GPFS<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
<div name="messageBodySection" class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">We do run Singularity + GPFS, on our production HPC clusters.
<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">Most of the time things are fine without any issues.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">However, i do see a significant performance loss when running some applications on singularity containers with GPFS.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">As of now, the applications that have severe performance issues with singularity on GPFS - seem to be because of “mmap io”. (Deep learning applications)<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">When i run the same application on bare metal, they seem to have a huge difference in GPFS IO when compared to running on singularity containers.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">I am yet to raise a PMR about this with IBM.<u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">I have not seen performance degradation for any other kind of IO, but i am not sure.<u class=""></u><u class=""></u></span></p>
</div>
</div>
<div name="messageSignatureSection" class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><br class="">
Regards,<br class="">
Lohit<u class=""></u><u class=""></u></span></p>
</div>
<div name="messageReplySection" class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><br class="">
On Apr 26, 2018, 10:35 AM -0400, Nathan Harper <<a href="mailto:nathan.harper@cfms.org.uk" target="_blank" class="">nathan.harper@cfms.org.uk</a>>, wrote:<br class="">
<br class="">
<u class=""></u><u class=""></u></span></p>
<blockquote style="border:none;border-left:solid #1abc9c 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt" class="">
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">We are running on a test system at the moment, and haven't run into any issues yet, but so far it's only been 'hello world' and running FIO.
<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">I'm interested to hear about experience with MPI-IO within Singularity.<u class=""></u><u class=""></u></span></p>
</div>
</div>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><u class=""></u> <u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">On 26 April 2018 at 15:20, Oesterlin, Robert <<a href="mailto:Robert.Oesterlin@nuance.com" target="_blank" class="">Robert.Oesterlin@nuance.com</a>> wrote:<u class=""></u><u class=""></u></span></p>
<blockquote style="border:none;border-left:solid #e67e22 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:3.75pt;margin-top:3.75pt;margin-right:3.75pt;margin-bottom:3.75pt" class="">
<div class="">
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">Anyone (including IBM) doing any work in this area? I would appreciate hearing from you.<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-family:"Arial",sans-serif" class=""> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">Bob Oesterlin<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">Sr Principal Storage Engineer, Nuance<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""> <u class=""></u><u class=""></u></span></p>
</div>
</div><p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><br class="">
______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=OTGc6UhyurUT-JrFZxFuLp_VKCTGGknEbzjMSKUa-QQ&e=" target="_blank" class="">
spectrumscale.org</a><br class="">
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=hPDughTsSwQUdR8KHv4gxXMumoGw3jXLgGGXG1MQrJM&e=" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><u class=""></u><u class=""></u></span></p>
</blockquote>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><br class="">
<br clear="all" class="">
<u class=""></u><u class=""></u></span></p>
<div class=""><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class=""><u class=""></u> <u class=""></u></span></p>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">--<u class=""></u><u class=""></u></span></p>
<div class="">
<div class=""><p class="MsoNormal" style="background:white"><b class=""><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222" class="">Nathan</span></b><b class=""><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#515058" class=""> </span></b><b class=""><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222" class="">Harper</span></b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#00aeff" class=""> // </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#222222" class="">IT
Systems Lead</span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><u class=""></u> <u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__cfms.org.uk_news-2Devents_events_2018_july_farnborough-2Dinternational-2Dairshow-2D2018_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=SAiH6rxM-6WYWwUBRecbBS9Yr7f4adg4GizgVJXIAqI&e=" target="_blank" class=""><span style="color:#222222;text-decoration:none" class=""><span style="color:blue;border:solid windowtext 1.0pt;padding:0in" class=""><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="m_-3404058168981204889_x0000_i1026" alt="Image removed by sender." class=""></span></span></a><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><br class="">
</span><strong class=""><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">e: </span></strong><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><a href="mailto:nathan.harper@cfms.org.uk" target="_blank" class=""><span style="color:#515058" class="">nathan.harper@cfms.org.uk</span></a> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><wbr class=""> </span><b class=""><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">t</span></b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">:</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> 0117
906 1104 </span><b class=""><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">m</span></b><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">:</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class="">
0787 551 0891 </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class=""> <strong class=""><span style="font-family:"Arial",sans-serif" class="">w: </span></strong></span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cfms.org.uk_&d=DwMFaQ&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=LPDewt1Z4o9eKc86MXmhqX-45Cz1yz1ylYELF9olLKU&m=_6XPyvEYCL-PnMjKsOcjeRy1H19CJ1ujTFPrAEwWnLo&s=KAbz_Md-BsCIrN6LDOb6Yc7sriwSq5IeO4aaxRGQfjc&e=" target="_blank" class=""><span style="color:#515058" class="">www.cfms.org.uk</span></a> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class=""> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058" class="">CFMS Services Ltd</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class=""> // </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058" class="">Bristol
& Bath Science Park</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058" class="">Dirac
Crescent</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058" class="">Emersons
Green</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058" class="">Bristol</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#00aeff" class="">//</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#515058" class="">BS16
7FR</span><span style="font-size:7.5pt;font-family:"Arial",sans-serif;color:#222222" class=""> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222;border:solid windowtext 1.0pt;padding:0in" class=""><img border="0" width="32" height="32" style="width:.3333in;height:.3333in" id="m_-3404058168981204889_x0000_i1025" alt="Image removed by sender." class=""></span><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222" class=""> </span><span style="font-size:10.5pt;font-family:"Arial",sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
<div class=""><p class="MsoNormal" style="background:white"><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:#515058" class="">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:#222222" class=""> <br class="">
</span><span style="font-size:7.5pt;font-family:"Verdana",sans-serif;color:#515058" class="">CFMS Services Ltd registered office // 43 Queens Square // Bristol // BS1 4QP</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#222222" class=""><u class=""></u><u class=""></u></span></p>
</div>
</div>
</div><p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial",sans-serif" class="">______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><u class=""></u><u class=""></u></span></p>
</blockquote>
</div>
</div>
</div></blockquote><blockquote type="cite" class=""><div class=""><span class="">______________________________<wbr class="">_________________</span><br class=""><span class="">gpfsug-discuss mailing list</span><br class=""><span class="">gpfsug-discuss at <a href="http://spectrumscale.org/" target="_blank" class="">spectrumscale.org</a></span><br class=""><span class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a></span><br class=""></div></blockquote></div></div></div><br class="">______________________________<wbr class="">_________________<br class="">
gpfsug-discuss mailing list<br class="">
gpfsug-discuss at <a href="http://spectrumscale.org/" rel="noreferrer" target="_blank" class="">spectrumscale.org</a><br class="">
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank" class="">http://gpfsug.org/mailman/<wbr class="">listinfo/gpfsug-discuss</a><br class="">
<br class=""></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><b style="font-family:Arial;font-size:small" class="">Nathan<span style="color:rgb(81,80,88)" class=""> </span>Harper</b><span style="font-family:Arial;font-size:small;color:rgb(0,174,255)" class=""> // </span><font size="2" style="font-family:Arial" class="">IT Systems Lead</font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><font size="2" style="font-family:Arial" class=""><br class=""></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><a href="https://cfms.org.uk/news-events/events/2018/july/farnborough-international-airshow-2018/" target="_blank" class=""><img style="width:500px;height:105px" src="https://cfms.org.uk/media/267028/fia-email-sig-blue-dark-3.jpg" class=""></a></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><br class=""><font face="Arial" size="1" class=""><strong class=""><span style="color:rgb(0,174,255)" class="">e: </span></strong><a href="mailto:nathan.harper@cfms.org.uk" style="color:rgb(81,80,88)" target="_blank" class="">nathan.harper@cfms.org.uk</a> <span style="color:rgb(0,174,255)" class=""> </span><font color="#222222" class=""> </font><font color="#00aeff" class=""><b class="">t</b>:</font><font color="#222222" class=""> 0117 906 1104</font> <font color="#222222" class=""> </font><font color="#00aeff" class=""><b class="">m</b>:</font><font color="#222222" class=""> 0787 551 0891 </font><span style="color:rgb(0,174,255)" class=""> <strong class="">w: </strong></span><a href="http://www.cfms.org.uk/" rel="noreferrer" style="color:rgb(81,80,88)" target="_blank" class="">www.cfms.org.uk</a> <span style="color:rgb(0,174,255)" class=""> </span></font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><font face="Arial" size="1" class=""><span style="color:rgb(81,80,88)" class="">CFMS Services Ltd</span><span style="color:rgb(0,174,255)" class=""> // </span><span style="color:rgb(81,80,88)" class="">Bristol & Bath Science Park</span> <span style="color:rgb(0,174,255)" class="">//</span> <span style="color:rgb(81,80,88)" class="">Dirac Crescent</span> <span style="color:rgb(0,174,255)" class="">//</span> <span style="color:rgb(81,80,88)" class="">Emersons Green</span> <span style="color:rgb(0,174,255)" class="">//</span> <span style="color:rgb(81,80,88)" class="">Bristol</span> <span style="color:rgb(0,174,255)" class="">//</span> <span style="color:rgb(81,80,88)" class="">BS16 7FR</span> </font></div><div style="color:rgb(34,34,34);background-color:rgb(255,255,255)" class=""><font face="Verdana" style="font-size:small" class=""><img src="http://cfms.org.uk/images/logo.png" class=""> </font></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;background-color:rgb(255,255,255)" class=""><span style="font-family:Verdana" class=""><span style="color:rgb(81,80,88);font-size:xx-small" class="">CFMS Services Ltd is registered in England and Wales No 05742022 - a subsidiary of CFMS Ltd</span> <br class=""><span style="color:rgb(81,80,88);font-size:xx-small" class="">CFMS Services Ltd registered office // </span></span><font color="#515058" face="Verdana" class=""><span style="font-size:xx-small" class="">43 Queens Square // Bristol // BS1 4QP</span></font></div></div>
</div>
_______________________________________________<br class="">gpfsug-discuss mailing list<br class="">gpfsug-discuss at <a href="http://spectrumscale.org" class="">spectrumscale.org</a><br class=""><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" class="">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br class=""></div></blockquote></div><br class=""></div></body></html>