<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Trebuchet MS";
panose-1:2 11 6 3 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:67306815;
mso-list-type:hybrid;
mso-list-template-ids:-233830298 -897424330 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Something I am thinking about doing is utilising the placement policy engine to insert custom metadata tags upon file creation, based on which fileset the creation occurs in. This might be to facilitate Research Data Management tasks that
could happen later in the data lifecycle.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am also thinking about allowing users to specify additional custom metadata tags (maybe through a fancy web interface) and also potentially give users control over creating new filesets (e.g. for scientists running new experiments). So…
pretend this is a placement policy on my GPFS driven data-ingest platform:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoPlainText">RULE 'RDMTEST'<o:p></o:p></p>
<p class="MsoPlainText"> SET POOL 'instruments’<o:p></o:p></p>
<p class="MsoPlainText"> FOR FILESET<o:p></o:p></p>
<p class="MsoPlainText" style="text-indent:36.0pt">('%GPFSRDM%10.01013%RDM%0ab34906-5357-4ca0-9d19-a470943db30a%RDM%8fc2395d-64c0-4ebd-8c71-0d2d34b3c1c0')<o:p></o:p></p>
<p class="MsoPlainText"> WHERE SetXattr<o:p></o:p></p>
<p class="MsoPlainText" style="text-indent:36.0pt">('user.rdm.parent','0ab34906-5357-4ca0-9d19-a470943db30a')<o:p></o:p></p>
<p class="MsoPlainText"> AND SetXattr<o:p></o:p></p>
<p class="MsoPlainText"> ('user.rdm.ingestor','8fc2395d-64c0-4ebd-8c71-0d2d34b3c1c0')<o:p></o:p></p>
<p class="MsoPlainText">RULE 'DEFAULT' SET POOL 'data'<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The fileset name can be meaningless (as far as the user is concerned), but would be linked somewhere nice that they recognise – say /gpfs/incoming/instrument1. The fileset, when it is created, would also be an AFM cache for its ‘home’ counterpart
which exists on a much larger (also GPFS driven) pool of storage… so that my metadata tags are preserved, you see.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This potentially user driven activity might look a bit like this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>User logs in to web interface and creates new experiment<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Filesets (system-generated names) are created on ‘home’ and ‘ingest’ file systems and linked into the directory namespace wherever the user specifies<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>AFM relationships are set up and established for the ingest (cache) fileset to write back to the AFM home fileset (probably Independent Writer mode)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>A set of ‘default’ policies are defined and installed on the cache file system to tag data for that experiment (the user can’t change these)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>The user now specifies additional metadata tags they want added to their experiment data (some of this might be captured through additional mandatory fields in the web form for instance)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>A policy for later execution by mmapplypolicy on the AFM home file system is created which looks for the tags generated at ingest-time and applies the extra user-defined tags<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There’s much more that would go on later in the lifecycle to take care of automated HSM tiering, data publishing, movement and cataloguing of data onto external non GPFS file systems, etc. but I won’t go in to it here. My GPFS related questions
are:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I install a placement policy into the file system, does the file system need to quiesce? My suspicion is yes, because the policy needs to be consistent on all nodes performing I/O, but I may be wrong.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What is the specific limitation for having a policy placement file no larger than 1MB?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal">Luke.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="background:white"><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">Luke Raimbach</span></b><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#595959;mso-fareast-language:EN-GB"></span></b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">Senior HPC Data and Storage Systems Engineer</span><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">The Francis Crick Institute</span><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">Gibbs Building</span><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">215 Euston Road</span><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">London NW1 2BE</span><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB"> </span><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">E: </span><a href="mailto:luke.raimbach@crick.ac.uk" target="_blank"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:blue;mso-fareast-language:EN-GB">luke.raimbach@crick.ac.uk</span></a><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal" style="background:white"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#595959;mso-fareast-language:EN-GB">W: </span><a href="http://www.crick.ac.uk/" target="_blank"><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:blue;mso-fareast-language:EN-GB">www.crick.ac.uk</span></a><b><span style="font-size:10.0pt;font-family:"Trebuchet MS","sans-serif";color:#212121;mso-fareast-language:EN-GB"><o:p></o:p></span></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p style="color:rgb(112,113,115);font-family: 'Trebuchet MS', 'Lucida Grande'; font-style: italic; font-size: 10pt;">
The Francis Crick Institute Limited is a registered charity in England and Wales no. 1140062 and a company registered in England and Wales no. 06885462, with its registered office at 215 Euston Road, London NW1 2BE.
</p>
</body>
</html>