<font size=2 face="sans-serif">Hi</font>
<br>
<br><font size=2 face="sans-serif">not going to mention much on DDN setups
but first thing that makes my eyes blurry a bit is</font>
<br>
<br><font size=3>minReleaseLevel 4.2.0.1</font>
<br>
<br><font size=2 face="sans-serif">when you mention your whole cluster
is already on 4.2.3</font>
<br><font size=2 face="sans-serif">--<br>
Ystävällisin terveisin / Kind regards / Saludos cordiales / Salutations<br>
Luis Bolinches<br>
Consultant IT Specialist<br>
Mobile Phone: +358503112585<br>
</font><a href="https://www.youracclaim.com/user/luis-bolinches"><font size=2 face="sans-serif">https://www.youracclaim.com/user/luis-bolinches</font></a><font size=2 face="sans-serif"><br>
<br>
"If you always give you will always have" -- Anonymous</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">John Hanks <griznog@gmail.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@spectrumscale.org></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">14/02/2018 17:22</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Odd behavior with cat followed by grep.</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@spectrumscale.org</font>
<br>
<hr noshade>
<br>
<br>
<br><font size=3>Hi Valdis,</font>
<br>
<br><font size=3>I tired with the grep replaced with 'ls -ls' and 'md5sum',
I don't think this is a data integrity issue, thankfully:</font>
<br>
<br><font size=3>$ ./pipetestls.sh </font>
<br><font size=3>256 -rw-r--r-- 1 39073 3001 530721 Feb 14 07:16 /srv/gsfs0/projects/pipetest.tmp.txt</font>
<br><font size=3>0 -rw-r--r-- 1 39073 3953 530721 Feb 14 07:16 /home/griznog/pipetest.tmp.txt</font>
<br>
<br><font size=3>$ ./pipetestmd5.sh </font>
<br><font size=3>15cb81a85c9e450bdac8230309453a0a /srv/gsfs0/projects/pipetest.tmp.txt</font>
<br><font size=3>15cb81a85c9e450bdac8230309453a0a /home/griznog/pipetest.tmp.txt</font>
<br>
<br><font size=3>And replacing grep with 'file' even properly sees the
files as ASCII:</font>
<br><font size=3>$ ./pipetestfile.sh </font>
<br><font size=3>/srv/gsfs0/projects/pipetest.tmp.txt: ASCII text, with
very long lines</font>
<br><font size=3>/home/griznog/pipetest.tmp.txt: ASCII text, with very
long lines</font>
<br>
<br><font size=3>I'll poke a little harder at grep next and see what the
difference in strace of each reveals.</font>
<br>
<br><font size=3>Thanks,</font>
<br>
<br><font size=3>jbh</font>
<br>
<br>
<br>
<br>
<br><font size=3>On Wed, Feb 14, 2018 at 7:08 AM, <</font><a href=mailto:valdis.kletnieks@vt.edu target=_blank><font size=3 color=blue><u>valdis.kletnieks@vt.edu</u></font></a><font size=3>>
wrote:</font>
<br><font size=3>On Wed, 14 Feb 2018 06:20:32 -0800, John Hanks said:<br>
<br>
> # ls -aln /srv/gsfs0/projects/pipetest.tmp.txt $HOME/pipetest.tmp.txt<br>
> -rw-r--r-- 1 39073 3953 530721 Feb 14 06:10 /home/griznog/pipetest.tmp.txt<br>
> -rw-r--r-- 1 39073 3001 530721 Feb 14 06:10<br>
> /srv/gsfs0/projects/pipetest.tmp.txt<br>
><br>
> We can "fix" the user case that exposed this by not using
a temp file or<br>
> inserting a sleep, but I'd still like to know why GPFS is behaving
this way<br>
> and make it stop.<br>
<br>
May be related to replication, or other behind-the-scenes behavior.<br>
<br>
Consider this example - 4.2.3.6, data and metadata replication both<br>
set to 2, 2 sites 95 cable miles apart, each is 3 Dell servers with a full<br>
fiberchannel mesh to 3 Dell MD34something arrays.<br>
<br>
% dd if=/dev/zero bs=1k count=4096 of=sync.test; ls -ls sync.test; sleep
5; ls -ls sync.test; sleep 5; ls -ls sync.test<br>
4096+0 records in<br>
4096+0 records out<br>
4194304 bytes (4.2 MB) copied, 0.0342852 s, 122 MB/s<br>
2048 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test<br>
8192 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test<br>
8192 -rw-r--r-- 1 root root 4194304 Feb 14 09:35 sync.test<br>
<br>
Notice that the first /bin/ls shouldn't be starting until after the dd
has<br>
completed - at which point it's only allocated half the blocks needed to
hold<br>
the 4M of data at one site. 5 seconds later, it's allocated the blocks
at both<br>
sites and thus shows the full 8M needed for 2 copies.<br>
<br>
I've also seen (but haven't replicated it as I write this) a small file
(4-8K<br>
or so) showing first one full-sized block, then a second full-sized block,
and<br>
then dropping back to what's needed for 2 1/32nd fragments. That
had me<br>
scratching my head<br>
<br>
Having said that, that's all metadata fun and games, while your case<br>
appears to have some problems with data integrity (which is a whole lot<br>
scarier). It would be *really* nice if we understood the problem
here.<br>
<br>
The scariest part is:<br>
<br>
> The first grep | wc -l returns 1, because grep outputs "Binary
file /path/to/<br>
> gpfs/mount/test matches"<br>
<br>
which seems to be implying that we're failing on semantic consistency.<br>
Basically, your 'cat' command is completing and closing the file, but then
a<br>
temporally later open of the same find is reading something other that
only the<br>
just-written data. My first guess is that it's a race condition similar
to the<br>
following: The cat command is causing a write on one NSD server, and the
first<br>
grep results in a read from a *different* NSD server, returning the data
that<br>
*used* to be in the block because the read actually happens before the
first<br>
NSD server actually completes the write.<br>
<br>
It may be interesting to replace the grep's with pairs of 'ls -ls / dd'
commands to grab the<br>
raw data and its size, and check the following:<br>
<br>
1) does the size (both blocks allocated and logical length) reported by<br>
ls match the amount of data actually read by the dd?<br>
<br>
2) Is the file length as actually read equal to the written length, or
does it<br>
overshoot and read all the way to the next block boundary?<br>
<br>
3) If the length is correct, what's wrong with the data that's telling
grep that<br>
it's a binary file? ( od -cx is your friend here).<br>
<br>
4) If it overshoots, is the remainder all-zeros (good) or does it return
semi-random<br>
"what used to be there" data (bad, due to data exposure issues)?<br>
<br>
(It's certainly not the most perplexing data consistency issue I've hit
in 4 decades - the<br>
winner *has* to be a intermittent data read corruption on a GPFS 3.5 cluster
that<br>
had us, IBM, SGI, DDN, and at least one vendor of networking gear all chasing
our<br>
tails for 18 months before we finally tracked it down. :)<br>
<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at </font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__spectrumscale.org&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=_UFKMxNklx_00YDdSlmEr9lCvnUC9AWFsTVbTn6yAr4&s=Rkt8qBEn82ZCRRabJYJG-8LACStOP4Yd9ZzOTsZEMrg&e=" target=_blank><font size=3 color=blue><u>spectrumscale.org</u></font></a><font size=3 color=blue><u><br>
</u></font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=_UFKMxNklx_00YDdSlmEr9lCvnUC9AWFsTVbTn6yAr4&s=JUVyUiTIfln67di06lb-hvwpA8207JNkioGxY1ayAlE&e=" target=_blank><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a><font size=3><br>
</font>
<br><tt><font size=2>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at spectrumscale.org<br>
</font></tt><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=_UFKMxNklx_00YDdSlmEr9lCvnUC9AWFsTVbTn6yAr4&s=JUVyUiTIfln67di06lb-hvwpA8207JNkioGxY1ayAlE&e="><tt><font size=2>https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1mZ896psa5caYzBeaugTlc7TtRejJp3uvKYxas3S7Xc&m=_UFKMxNklx_00YDdSlmEr9lCvnUC9AWFsTVbTn6yAr4&s=JUVyUiTIfln67di06lb-hvwpA8207JNkioGxY1ayAlE&e=</font></tt></a><tt><font size=2><br>
</font></tt>
<br>
<br><font size=2 face="sans-serif"><br>
<br>
Ellei edellä ole toisin mainittu: / Unless stated otherwise above:<br>
Oy IBM Finland Ab<br>
PL 265, 00101 Helsinki, Finland<br>
Business ID, Y-tunnus: 0195876-3 <br>
Registered in Finland<br>
</font>