<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello to all!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m pleased to have joined the GPFS UG mailing list, I’m experimenting with GPFS on zLinux running in z/VM on a z13 mainframe. I work for the UK Met Office in the GPCS team (general purpose compute service/mainframe team) and I’m based
in Exeter, Devon.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve joined with a specific question to ask, in short: how can I automate sending files to a cloud object store as they arrive in GPFS and keep a copy of the file in GPFS?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The longer spiel is this: We have a HPC that throws out a lot of NetCDF files via FTP for use in forecasts. We’re currently undergoing a change in working practice, so that data processing is beginning to be done in the cloud. At the same
time we’re also attempting to de-duplicate the data being sent from the HPC by creating one space to receive it and then have consumers use it or send it on as necessary from there. The data is in terabytes a day sizes, and the timeliness of it’s arrival to
systems is fairly important (forecasts cease to be forecasts if they’re too late).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We’re using zLinux because the mainframe already receives much of the data from the HPC and has access to a SAN with SSD storage, has the right network connections it needs and generally seems the least amount of work to put something in
place.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Getting a supported clustered filesystem on zLinux is tricky, but GPFS fits the bill and having hardware, storage, OS and filesystem from one provider (IBM) should hopefully save some headaches.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We’re using Amazon as our cloud provider, and have 2x10GB direct links to their London data centre with a ping of about 15ms, so fairly low latency. The developers using the data want it in s3 so they can access it from server-less environments
and won’t need to have ec2 instances loitering to look after the data.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We were initially interested in using mmcloudgateway/cloud data sharing to send the data, but it’s not available for s390x (only x86_64), so I’m now looking at setting up a external storage pool for talking to s3 and then having some kind
of ilm soft quota trigger to send the data once enough of it has arrived, but I’m still exploring options. Options such as asking the user group of experienced folks what they think is best!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So, any help or advice would be greatly appreciated!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Peter Chase<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">GPCS Team</span><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Met Office</span></b><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> FitzRoy Road Exeter Devon EX1 3PB United Kingdom<br>
Email: <a href="mailto:peter.chase@metoffice.gov.uk" title="blocked::mailto:peter.chase@metoffice.gov.uk">
<span style="color:blue">peter.chase@metoffice.gov.uk</span></a> Website: <a href="http://www.metoffice.gov.uk/" title="http://www.metoffice.gov.uk/">
<span style="color:blue">www.metoffice.gov.uk</span></a> </span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>