<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt" ><div dir="ltr" >Kevin,</div>
<div dir="ltr" > </div>
<div dir="ltr" >I'm repeating the relevant portion of your story here:</div>
<div dir="ltr" > </div>
<div dir="ltr" >-----------</div>
<div dir="ltr" ><div>"In the GPFS 4.2 Advanced Administration Guide, it states, “If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.”</div>
<div> </div>
<div>This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don’t remember for sure) later. I was under the obviously mistaken impression that the “first” data storage pool was the system pool, but that is clearly not correct.</div>
<div> </div>
<div>So my first question is, what is the definition of “the first storage pool?” and my second question is, can the documentation be updated with the answer to my first question since it’s clearly ambiguous as written now? Thanks…"</div>
<div>-----------</div>
<div> </div>
<div>1. If you run "mmlspolicy YOUR_FS -L" you'll see a message indicating where your data will be placed.</div>
<div> </div>
<div>One-disk FS with a system pool:</div>
<div> </div>
<div># mmlsdisk testfs<br>Disk driver sector failure holds holds storage<br>name type size group metadata data status availability pool<br>------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------<br>nsd4 nsd 512 -1 yes yes ready up system <br>#11:36:59# hs22n86:~ # mmlspolicy testfs -L<br>GPFS: 6027-793 No policy file was installed for file system 'testfs'.<br>Data will be stored in pool 'system'.</div>
<div> </div>
<div>Adding another disk into a 'gold' pool:</div>
<div> </div>
<div>#11:38:12# hs22n86:~ # mmlsdisk testfs<br>disk driver sector failure holds holds storage<br>name type size group metadata data status availability pool<br>------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------<br>nsd4 nsd 512 -1 yes yes ready up system <br>nsd5 nsd 512 -1 no yes ready up gold </div>
<div><div>#11:38:22# hs22n86:~ # mmlspolicy testfs -L<br>GPFS: 6027-793 No policy file was installed for file system 'testfs'.<br>Data will be stored in pool 'gold'.</div>
<div> </div></div>
<div>2. Focus first not on the "first" but on the "data". The system pool is a metadata pool, and all other pools are data pools. In the absence of a storage policy, any data you write into the file system will be placed in the first [1] of your data (read: non-system) pools.</div>
<div>3. The story of your file system is as follows:</div>
<div> </div>
<div>Once upon a time, there was only a system pool. Lacking any alternative, and with non-metadataOnly disks, all data was stored in it. Then you defined disks in a data pool. As of GPFS 4.1 (?), in the absence of a storage policy GPFS began automatically placing data in the first data pool [2]. This leaves you with some user data in the system pool and some user data in the first data pool.</div>
<div> </div>
<div>Notes:</div>
<div>[1] This is presumably the 'first-defined', but check mmlspolicy -L if you're not sure.</div></div>
<div dir="ltr" >[2] Prior to the (4.1?) release, the default behavior often ended up with I/O errors on trying to write into a not-quite-configured file system. This produced enough complaints from both internal and external users that we opted to just place the data somewhere reasonable when the FS seemed not-quite-configured. Unfortunately the result might seem a bit confusing if you're extending an existing FS with new dataOnly storage pools.</div>
<div dir="ltr" > </div>
<div dir="ltr" >Hope this helps!</div>
<div dir="ltr" > </div>
<div dir="ltr" >Jamie</div>
<div dir="ltr" > </div>
<div dir="ltr" >Jamie Davis<br>GPFS Functional Verification Test (FVT)<br>jamiedavis@us.ibm.com</div>
<div dir="ltr" > </div>
<div dir="ltr" > </div>
<blockquote data-history-content-modified="1" dir="ltr" style="border-left:solid #aaaaaa 2px; margin-left:5px; padding-left:5px; direction:ltr; margin-right:0px" >----- Original message -----<br>From: "Buterbaugh, Kevin L" <Kevin.Buterbaugh@Vanderbilt.Edu><br>Sent by: gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [gpfsug-discuss] Initial file placement<br>Date: Fri, Jun 17, 2016 11:19 AM<br> <br><!--Notes ACF
<meta http-equiv="Content-Type" content="text/html; charset=utf8" >--> Hi yet again all,
<div> </div>
<div>Well, this has turned out to be an enlightening and surprising morning in GPFS land… </div>
<div> </div>
<div>What prompted my question below is this … I am looking to use the new QoS features in GPFS 4.2. I have QoS enabled and am trying to get a baseline of IOPs so that I can determine how much I want to assign to the maintenance class (currently both maintenance and other are set to unlimited). To do this, I fired off a bonnie++ test from each of my NSD servers.</div>
<div> </div>
<div>The filesystem in question has two storage pools, the system pool and the capacity pool. The system pool is comprised of a couple of metadata only disks (SSD-based RAID 1 mirrors) and several data only disks (spinning HD-based RAID 6), while the capacity pool is comprised exclusively of data only disks (RAID 6).</div>
<div> </div>
<div>When the bonnie++’s were creating, reading, and rewriting the big file they create I was quite surprised to see mmlsqos show higher IOP’s on the capacity pool than the system pool by a factor of 10! As I was expecting those files to be being written to the system pool, this was quite surprising to me. Once I found the mmlsattr command, I ran it on one of the files being created and saw that it was indeed assigned to the capacity pool. The bonnie++’s finished before I could check the other files.</div>
<div> </div>
<div>I don’t have any file placement policies in effect for this filesystem, only file migration policies (each weekend any files in the system pool with an atime > 60 days get moved to the capacity pool and any files in the capacity pool with an atime < 60 days get moved to the system pool).</div>
<div> </div>
<div>In the GPFS 4.2 Advanced Administration Guide, it states, “If a GPFS file system does not have a placement policy installed, all the data is stored in the first data storage pool.”</div>
<div> </div>
<div>This filesystem was initially created in 2010 and at that time consisted only of the system pool. The capacity pool was not created until some years (2014? 2015? don’t remember for sure) later. I was under the obviously mistaken impression that the “first” data storage pool was the system pool, but that is clearly not correct.</div>
<div> </div>
<div>So my first question is, what is the definition of “the first storage pool?” and my second question is, can the documentation be updated with the answer to my first question since it’s clearly ambiguous as written now? Thanks…</div>
<div> </div>
<div>Kevin</div>
<div> </div>
<div><div><div><blockquote type="cite" ><div>On Jun 17, 2016, at 9:29 AM, Buterbaugh, Kevin L <<a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" >Kevin.Buterbaugh@Vanderbilt.Edu</a>> wrote:</div>
<div><div>Hi All,
<div> </div>
<div>I am aware that with the mmfileid command I can determine which files have blocks on a given NSD. But is there a way to query a particular file to see which NSD(s) is has blocks on? Thanks in advance…</div>
<div> </div>
<div>Kevin</div>
<div>
<div><div>—</div>
<div>Kevin Buterbaugh - Senior System Administrator</div>
<div>Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" >Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div> </div></div></div></div>_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank" >spectrumscale.org</a><br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></div></blockquote></div>
<div><div>—</div>
<div>Kevin Buterbaugh - Senior System Administrator</div>
<div>Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" target="_blank" >Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div> </div></div></div></div>
<div><font face="Default Monospace,Courier New,Courier,monospace" size="2" >_______________________________________________<br>gpfsug-discuss mailing list<br>gpfsug-discuss at spectrumscale.org<br><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank" >http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></font></div></blockquote>
<div dir="ltr" > </div></div><BR>