<div dir="ltr">Straces are interesting, but don't immediately open my eyes:<div><br></div><div>strace of grep on NFS (works as expected)</div><div><br></div><div><div>openat(AT_FDCWD, "/home/griznog/pipetest.tmp.txt", O_RDONLY) = 3</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=530721, ...}) = 0</div><div>ioctl(3, TCGETS, 0x7ffe2c26b0b0) = -1 ENOTTY (Inappropriate ioctl for device)</div><div>read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768</div><div>lseek(3, 32768, SEEK_HOLE) = 530721</div><div>lseek(3, 32768, SEEK_SET) = 32768</div><div>fstat(1, {st_mode=S_IFREG|0644, st_size=5977, ...}) = 0</div><div>mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3bf6c43000</div><div>write(1, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 8192chr1 </div></div><div><br></div><div>strace on GPFS (thinks file is binary)</div><div><div><br></div><div>openat(AT_FDCWD, "/srv/gsfs0/projects/pipetest.tmp.txt", O_RDONLY) = 3</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=530721, ...}) = 0</div><div>ioctl(3, TCGETS, 0x7ffc9b52caa0) = -1 ENOTTY (Inappropriate ioctl for device)</div><div>read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768</div><div>lseek(3, 32768, SEEK_HOLE) = 262144</div><div>lseek(3, 32768, SEEK_SET) = 32768</div><div>fstat(1, {st_mode=S_IFREG|0644, st_size=6011, ...}) = 0</div><div>mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd45ee88000</div><div>close(3) = 0</div><div>write(1, "Binary file /srv/gsfs0/projects/"..., 72Binary file /srv/gsfs0/projects/levinson/xwzhu/pipetest.tmp.txt matches</div><div>) = 72</div></div><div><br></div><div>Do the lseek() results indicate that the grep on the GPFS mounted version thinks the file is a sparse file? For comparison I strace'd md5sum in place of the grep and it does not lseek() with SEEK_HOLE, it's access in both cases look identical, like:</div><div><br></div><div><div>open("/home/griznog/pipetest.tmp.txt", O_RDONLY) = 3</div><div>fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0</div><div>fstat(3, {st_mode=S_IFREG|0644, st_size=530721, ...}) = 0</div><div>mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb7d2c2b000</div><div>read(3, "chr1\t43452652\t43452652\tL1HS\tlib1"..., 32768) = 32768</div><div>...[reads clipped]...</div><div>read(3, "", 24576) = 0<br></div><div>lseek(3, 0, SEEK_CUR) = 530721</div><div>close(3) = 0</div></div><div><br></div><div><br></div><div>jbh</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 14, 2018 at 9:51 AM, Aaron Knister <span dir="ltr"><<a href="mailto:aaron.s.knister@nasa.gov" target="_blank">aaron.s.knister@nasa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Just speculating here (also known as making things up) but I wonder if<br>
grep is somehow using the file's size in its determination of binary<br>
status. I also see mmap in the strace so maybe there's some issue with<br>
mmap where some internal GPFS buffer is getting truncated<br>
inappropriately but leaving a bunch of null values which gets returned<br>
to grep.<br>
<br>
-Aaron<br>
<span class=""><br>
On 2/14/18 10:21 AM, John Hanks wrote:<br>
> Hi Valdis,<br>
><br>
> I tired with the grep replaced with 'ls -ls' and 'md5sum', I don't think<br>
> this is a data integrity issue, thankfully:<br>
><br>
> $ ./pipetestls.sh <br>
> 256 -rw-r--r-- 1 39073 3001 530721 Feb 14 07:16<br>
> /srv/gsfs0/projects/pipetest.<wbr>tmp.txt<br>
> 0 -rw-r--r-- 1 39073 3953 530721 Feb 14 07:16 /home/griznog/pipetest.tmp.txt<br>
><br>
> $ ./pipetestmd5.sh <br>
> 15cb81a85c9e450bdac8230309453a<wbr>0a /srv/gsfs0/projects/pipetest.<wbr>tmp.txt<br>
> 15cb81a85c9e450bdac8230309453a<wbr>0a /home/griznog/pipetest.tmp.txt<br>
><br>
> And replacing grep with 'file' even properly sees the files as ASCII:<br>
> $ ./pipetestfile.sh <br>
> /srv/gsfs0/projects/pipetest.<wbr>tmp.txt: ASCII text, with very long lines<br>
> /home/griznog/pipetest.tmp.<wbr>txt: ASCII text, with very long lines<br>
><br>
> I'll poke a little harder at grep next and see what the difference in<br>
> strace of each reveals.<br>
><br>
> Thanks,<br>
><br>
> jbh<br>
><br>
><br>
><br>
><br>
> On Wed, Feb 14, 2018 at 7:08 AM, <<a href="mailto:valdis.kletnieks@vt.edu">valdis.kletnieks@vt.edu</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:valdis.kletnieks@vt.edu">valdis.kletnieks@vt.<wbr>edu</a>>> wrote:<br>
><br>
> On Wed, 14 Feb 2018 06:20:32 -0800, John Hanks said:<br>
><br>
> > # ls -aln /srv/gsfs0/projects/pipetest.<wbr>tmp.txt $HOME/pipetest.tmp.txt<br>
> > -rw-r--r-- 1 39073 3953 530721 Feb 14 06:10 /home/griznog/pipetest.tmp.txt<br>
> > -rw-r--r-- 1 39073 3001 530721 Feb 14 06:10<br>
> > /srv/gsfs0/projects/pipetest.<wbr>tmp.txt<br>
> ><br>
> > We can "fix" the user case that exposed this by not using a temp file or<br>
> > inserting a sleep, but I'd still like to know why GPFS is behaving this way<br>
> > and make it stop.<br>
><br>
> May be related to replication, or other behind-the-scenes behavior.<br>
><br>
> Consider this example - 4.2.3.6, data and metadata replication both<br>
> set to 2, 2 sites 95 cable miles apart, each is 3 Dell servers with<br>
> a full<br>
> fiberchannel mesh to 3 Dell MD34something arrays.<br>
><br>
> % dd if=/dev/zero bs=1k count=4096 of=sync.test; ls -ls sync.test;<br>
> sleep 5; ls -ls sync.test; sleep 5; ls -ls sync.test<br>
> 4096+0 records in<br>
> 4096+0 records out<br>
> 4194304 bytes (4.2 MB) copied, 0.0342852 s, 122 MB/s<br>
> 2048 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test<br>
> 8192 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test<br>
> 8192 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test<br>
><br>
> Notice that the first /bin/ls shouldn't be starting until after the<br>
> dd has<br>
> completed - at which point it's only allocated half the blocks<br>
> needed to hold<br>
> the 4M of data at one site. 5 seconds later, it's allocated the<br>
> blocks at both<br>
> sites and thus shows the full 8M needed for 2 copies.<br>
><br>
> I've also seen (but haven't replicated it as I write this) a small<br>
> file (4-8K<br>
> or so) showing first one full-sized block, then a second full-sized<br>
> block, and<br>
> then dropping back to what's needed for 2 1/32nd fragments. That had me<br>
> scratching my head<br>
><br>
> Having said that, that's all metadata fun and games, while your case<br>
> appears to have some problems with data integrity (which is a whole lot<br>
> scarier). It would be *really* nice if we understood the problem here.<br>
><br>
> The scariest part is:<br>
><br>
> > The first grep | wc -l returns 1, because grep outputs "Binary file /path/to/<br>
> > gpfs/mount/test matches"<br>
><br>
> which seems to be implying that we're failing on semantic consistency.<br>
> Basically, your 'cat' command is completing and closing the file,<br>
> but then a<br>
> temporally later open of the same find is reading something other<br>
> that only the<br>
> just-written data. My first guess is that it's a race condition<br>
> similar to the<br>
> following: The cat command is causing a write on one NSD server, and<br>
> the first<br>
> grep results in a read from a *different* NSD server, returning the<br>
> data that<br>
> *used* to be in the block because the read actually happens before<br>
> the first<br>
> NSD server actually completes the write.<br>
><br>
> It may be interesting to replace the grep's with pairs of 'ls -ls /<br>
> dd' commands to grab the<br>
> raw data and its size, and check the following:<br>
><br>
> 1) does the size (both blocks allocated and logical length) reported by<br>
> ls match the amount of data actually read by the dd?<br>
><br>
> 2) Is the file length as actually read equal to the written length,<br>
> or does it<br>
> overshoot and read all the way to the next block boundary?<br>
><br>
> 3) If the length is correct, what's wrong with the data that's<br>
> telling grep that<br>
> it's a binary file? ( od -cx is your friend here).<br>
><br>
> 4) If it overshoots, is the remainder all-zeros (good) or does it<br>
> return semi-random<br>
> "what used to be there" data (bad, due to data exposure issues)?<br>
><br>
> (It's certainly not the most perplexing data consistency issue I've<br>
> hit in 4 decades - the<br>
> winner *has* to be a intermittent data read corruption on a GPFS 3.5<br>
> cluster that<br>
> had us, IBM, SGI, DDN, and at least one vendor of networking gear<br>
> all chasing our<br>
> tails for 18 months before we finally tracked it down. :)<br>
><br>
> ______________________________<wbr>_________________<br>
> gpfsug-discuss mailing list<br>
</div></div>> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a> <<a href="http://spectrumscale.org" rel="noreferrer" target="_blank">http://spectrumscale.org</a>><br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
> <<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a>><br>
<span class="im HOEnZb">><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
><br>
<br>
</span><span class="HOEnZb"><font color="#888888">--<br>
Aaron Knister<br>
NASA Center for Climate Simulation (Code 606.2)<br>
Goddard Space Flight Center<br>
<a href="tel:%28301%29%20286-2776" value="+13012862776">(301) 286-2776</a><br>
</font></span><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/<wbr>listinfo/gpfsug-discuss</a><br>
</div></div></blockquote></div><br></div>