<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>So as Venkat says, AFM doesn't support using fallocate() to preallocate space.</div>
<div><br>
</div>
<div>So why aren't other people seeing this ... Well ...</div>
<div><br>
</div>
<div>We use EasyBuild to build our HPC cluster software including the compiler tool chains.</div>
<div>This enables the new linker ld.gold by default rather than the "old" ld.</div>
<div>Interestingly we don't seem to have seen this with C code being compiled, only fortran.</div>
<div>We can work around it by using the options to gfortran I mention below.</div>
<div><br>
</div>
<div>There is a mention to this limitation at:</div>
<div><a href="https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_afmlimitations.htm">https://www.ibm.com/support/knowledgecenter/en/STXKQY_4.2.3/com.ibm.spectrum.scale.v4r23.doc/bl1ins_afmlimitations.htm</a></div>
<div><br>
</div>
<div>We aren;t directly calling gpfs_prealloc, but I guess the linker is indirectly calling it by making a call to posix_fallocate.</div>
<div><br>
</div>
<div>I do have a new problem with AFM where the data written to the cache differs from that replicated back to home... I'm beginning to think I don't like the decision to use AFM! Given the data written back to HOME is corrupt, I think this is definitely PMR
time. But ... If you have Abaqus on you system and are using AFM, I'd be interested to see if someone else sees the same issue as us!</div>
<div><br>
</div>
<div>Simon</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span><<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a>> on behalf of Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>><br>
<span style="font-weight:bold">Reply-To: </span>"<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, 23 August 2017 at 14:01<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [gpfsug-discuss] AFM weirdness<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>I've got a PMR open about this ... Will email you the number directly.</div>
<div><br>
</div>
<div>Looking at the man page for ld.gold, it looks to set '<b style="color: rgb(51, 51, 51); font-family: inherit; font-size: 12px; orphans: 2; widows: 2;">--posix-fallocate'</b> by default. In fact, testing with '-Xlinker -no-posix-fallocate' does indeed make
the code compile.</div>
<div><br>
</div>
<div>Simon</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"<a href="mailto:vpuvvada@in.ibm.com">vpuvvada@in.ibm.com</a>" <<a href="mailto:vpuvvada@in.ibm.com">vpuvvada@in.ibm.com</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, 23 August 2017 at 13:36<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>" <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>>, Simon Thompson <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [gpfsug-discuss] AFM weirdness<br>
</div>
<div><br>
</div>
<div>
<div><font size="2" face="sans-serif">I believe this error is result of preallocation failure, but traces are needed to confirm this. AFM caching modes does not support preallocation of blocks (ex. using fallocate()). This feature is supported only in AFM
DR.</font><br>
<br>
<font size="2" face="sans-serif">~Venkat (<a href="mailto:vpuvvada@in.ibm.com">vpuvvada@in.ibm.com</a>)</font><br>
<br>
<br>
<br>
<font size="1" color="#5f5f5f" face="sans-serif">From: </font><font size="1" face="sans-serif">"Simon Thompson (IT Research Support)" <<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>></font><br>
<font size="1" color="#5f5f5f" face="sans-serif">To: </font><font size="1" face="sans-serif">gpfsug main discussion list <<a href="mailto:gpfsug-discuss@spectrumscale.org">gpfsug-discuss@spectrumscale.org</a>></font><br>
<font size="1" color="#5f5f5f" face="sans-serif">Date: </font><font size="1" face="sans-serif">08/23/2017 03:48 PM</font><br>
<font size="1" color="#5f5f5f" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [gpfsug-discuss] AFM weirdness</font><br>
<font size="1" color="#5f5f5f" face="sans-serif">Sent by: </font><font size="1" face="sans-serif"><a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a></font><br>
<hr noshade="">
<br>
<br>
<br>
<tt><font size="2">OK so I checked and if I run directly on the "AFM" FS in a different "non<br>
AFM" directory, it works fine, so its something AFM related ...<br>
<br>
Simon<br>
<br>
On 23/08/2017, 11:11, "<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a> on behalf<br>
of Simon Thompson (IT Research Support)"<br>
<<a href="mailto:gpfsug-discuss-bounces@spectrumscale.org">gpfsug-discuss-bounces@spectrumscale.org</a> on behalf of<br>
<a href="mailto:S.J.Thompson@bham.ac.uk">S.J.Thompson@bham.ac.uk</a>> wrote:<br>
<br>
>We're using an AFM cache from our HPC nodes to access data in another GPFS<br>
>cluster, mostly this seems to be working fine, but we've just come across<br>
>an interesting problem with a user using gfortran from the GCC 5.2.0<br>
>toolset.<br>
><br>
>When linking their code, they get a "no space left on device" error back<br>
>from the linker. If we do this on a node that mounts the file-system<br>
>directly (I.e. Not via AFM cache), then it works fine.<br>
><br>
>We tried with GCC 4.5 based tools and it works OK, but the difference<br>
>there is that 4.x uses ld and 5x uses ld.gold.<br>
><br>
>If we strike the ld.gold when using AFM, we see:<br>
><br>
>stat("program", {st_mode=S_IFREG|0775, st_size=248480, ...}) = 0<br>
>unlink("program") = 0<br>
>open("program", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0777) = 30<br>
>fstat(30, {st_mode=S_IFREG|0775, st_size=0, ...}) = 0<br>
>fallocate(30, 0, 0, 248480) = -1 ENOSPC (No space left on<br>
>device)<br>
><br>
><br>
><br>
>Vs when running directly on the file-system:<br>
>stat("program", {st_mode=S_IFREG|0775, st_size=248480, ...}) = 0<br>
>unlink("program") = 0<br>
>open("program", O_RDWR|O_CREAT|O_TRUNC|O_CLOEXEC, 0777) = 30<br>
>fstat(30, {st_mode=S_IFREG|0775, st_size=0, ...}) = 0<br>
>fallocate(30, 0, 0, 248480) = 0<br>
><br>
><br>
><br>
>Anyone seen anything like this before?<br>
><br>
>... Actually I'm about to go off and see if its a function of AFM, or<br>
>maybe something to do with the FS in use (I.e. Make a local directory on<br>
>the filesystem on the "AFM" FS and see if that works ...)<br>
><br>
>Thanks<br>
><br>
>Simon<br>
><br>
>_______________________________________________<br>
>gpfsug-discuss mailing list<br>
>gpfsug-discuss at spectrumscale.org<br>
></font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=UqTzoU-bx454OgyeB4f0Nrruvs7yYAxFutzIe2eKmnc&s=8E5opHyyAwomLS8kdxpvKCvf6sdKBLlfZvx6wDdaZy4&e="><tt><font size="2">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=UqTzoU-bx454OgyeB4f0Nrruvs7yYAxFutzIe2eKmnc&s=8E5opHyyAwomLS8kdxpvKCvf6sdKBLlfZvx6wDdaZy4&e=</font></tt></a><tt><font size="2"><br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=UqTzoU-bx454OgyeB4f0Nrruvs7yYAxFutzIe2eKmnc&s=8E5opHyyAwomLS8kdxpvKCvf6sdKBLlfZvx6wDdaZy4&e="><tt><font size="2">https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=92LOlNh2yLzrrGTDA7HnfF8LFr55zGxghLZtvZcZD7A&m=UqTzoU-bx454OgyeB4f0Nrruvs7yYAxFutzIe2eKmnc&s=8E5opHyyAwomLS8kdxpvKCvf6sdKBLlfZvx6wDdaZy4&e=</font></tt></a><tt><font size="2"><br>
<br>
</font></tt><br>
<br>
<br>
</div>
</div>
</span></div>
</div>
</span>
</body>
</html>