<div dir="ltr">Hi,<div><br></div><div>I have tried this before and I would like to temper your expectations.</div><div><br></div><div>If you use a placement policy to allow users to write any files into your "small" pool (e.g. by directory), they will get E_NOSPC when your small pool fills up. And they will be confused because they can't see the pool configuration, they just see a large filesystem with lots of space. I think there may now be an "overflow" policy but it will only work for new files, not if someone keeps writing into an existing file in an existing pool.</div><div><br></div><div>If you use a migration policy (even based on heat map) it is still a periodic scheduled data movement and not anything that happens "on the fly". Also, "fileheat" only gets updated at some interval anyway.</div><div><br></div><div>If you use a migration policy to move data between pools, you may starve users of I/O which will confuse your users because suddenly things are slow. I think there is now a QOS way to throttle your data migration. I guess it depends on how much of your disk I/O throughput is not used; if your disks are already churning, migrations will just slow everything down.</div><div><br></div><div>Think of it less like a cache layer and more like two separate storage locations. If a bunch of jobs want to read the same files from your big pool, it's probably faster to just have them read from the big pool directly rather than have some kind of prologue job to read the data from the big pool, write it into the small poool, then have the jobs read from the small pool.</div><div><br></div><div>Also, my experience was with pool ratios of like 10%/90%, yours is more like 2%/98%. However, mine were with write-heavy workloads (typical university environment with quickly growing capacity utilization).</div><div><br></div><div>Hope these anecdotes help. Also, it could be that things work a bit differently now in new versions.</div><div><br></div><div>Regards,</div><div>Alex</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 6, 2019 at 3:13 AM Jake Carroll <<a href="mailto:jake.carroll@uq.edu.au">jake.carroll@uq.edu.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Scale-folk.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I have an IBM ESS GH14S building block currently configured for my HPC workloads.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I've got about 1PB of /scratch filesystem configured in mechanical spindles via GNR and about 20TB of SSD/flash sitting in another GNR filesystem at the moment. My intention is to destroy that stand-alone flash filesystem eventually and use storage pools coupled
with GPFS policy to warm up workloads into that flash storage:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_storagepool.htm" id="gmail-m_-1885111439077405740LPNoLP864345" target="_blank">https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1adv_storagepool.htm</a><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
A little dated, but that kind of thing.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Does anyone have any experience in this space in using flash storage inside a pool with pre/post flight SLURM scripts to puppeteer GPFS policy to warm data up?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I had a few ideas for policy construction around file size, file count, file access intensity. Someone mentioned heat map construction and mmdiag --iohist to me the other day. Could use some background there.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
If anyone has any SLURM specific integration tips for the scheduler or pre/post flight bits for SBATCH, it'd be really very much appreciated.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
This array really does fly along and surpassed my expectations - but, I want to get the most out of it that I can for my users - and I think storage pool automation and good file placement management is going to be an important part of that.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thank you.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-jc</div>
</div>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div>