<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>(I'm slowly catching up on a backlog of e-mail, sorry for the
delayed reply).<br>
</p>
<p>Thanks, Sven. I recognize the complexity and appreciate your
explanation.<br>
</p>
<p>In my mind I had envisioned either the block integrity
information being stored as a new metadata structure or stored
leveraging T10-DIX/DIF (perhaps configurable on a per-pool basis)
to pass the checksums down to the RAID controller.<br>
</p>
I would quite like to run GNR as software on generic hardware and in
fact voted, along with 26 other customers, on an RFE
(<a class="moz-txt-link-freetext" href="https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090">https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090</a>)
requesting this but the request was declined. I think customers
spoke pretty loudly there and IBM gave it the kibosh.<br>
<br>
-Aaron<br>
<br>
<div class="moz-cite-prefix">On 06/30/2017 02:25 PM, Sven Oehme
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALssuR02VGRnrzZPfN+-A5kCpaOdFpDnC9ehr=WouJNf_2LhoA@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">
<p class="inbox-inbox-p1">end-to-end data integrity is very
important and the reason it hasn't been done in Scale is not
because its not important, its because its very hard to do
without impacting performance in a very dramatic way.<span
class="inbox-inbox-Apple-converted-space"> </span></p>
<p class="inbox-inbox-p1">imagine your raid controller blocksize
is 1mb and your filesystem blocksize is 1MB . if your
application does a 1 MB write this ends up being a perfect
full block , full track de-stage to your raid layer and
everything works fine and fast. as soon as you add checksum
support you need to add data somehow into this, means your 1MB
is no longer 1 MB but 1 MB+checksum.<span
class="inbox-inbox-Apple-converted-space"> </span></p>
<p class="inbox-inbox-p1">to store this additional data you have
multiple options, inline , outside the data block or some
combination ,the net is either you need to do more physical
i/o's to different places to get both the data and the
corresponding checksum or your per block on disc structure
becomes bigger than than what your application reads/or
writes, both put massive burden on the Storage layer as e.g. a
1 MB write will now, even the blocks are all aligned from the
application down to the raid layer, cause a read/modify/write
on the raid layer as the data is bigger than the physical
track size.<span class="inbox-inbox-Apple-converted-space"> </span></p>
<p class="inbox-inbox-p2">so to get end-to-end checksum in Scale
outside of ESS the best way is to get GNR as SW to run on
generic HW, this is what people should vote for as RFE if they
need that functionality. beside end-to-end checksums you get
read/write cache and acceleration , fast rebuild and many
other goodies as a added bonus. <br>
</p>
<p class="inbox-inbox-p2">Sven<br>
</p>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Jun 30, 2017 at 10:53 AM Aaron Knister
<<a href="mailto:aaron.s.knister@nasa.gov"
moz-do-not-send="true">aaron.s.knister@nasa.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">In fact the
answer was quite literally "no":<br>
<br>
<a
href="https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523</a><br>
(the RFE was declined and the answer was that the "function is
already<br>
available in GNR environments").<br>
<br>
Regarding GNR, see this RFE request<br>
<a
href="https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090</a><br>
requesting the use of GNR outside of an ESS/GSS environment.
It's<br>
interesting to note this is the highest voted Public RFE for
GPFS that I<br>
can see, at least. It too was declined.<br>
<br>
-Aaron<br>
<br>
On 6/30/17 1:41 PM, Aaron Knister wrote:<br>
> Thanks Olaf, that's good to know (and is kind of what I
suspected). I've<br>
> requested a number of times this capability for those of
us who can't<br>
> use or aren't using GNR and the answer is effectively
"no". This<br>
> response is curious to me because I'm sure IBM doesn't
believe that data<br>
> integrity is only important and of value to customers who
purchase their<br>
> hardware *and* software.<br>
><br>
> -Aaron<br>
><br>
> On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser <<a
href="mailto:olaf.weiser@de.ibm.com" target="_blank"
moz-do-not-send="true">olaf.weiser@de.ibm.com</a><br>
> <mailto:<a href="mailto:olaf.weiser@de.ibm.com"
target="_blank" moz-do-not-send="true">olaf.weiser@de.ibm.com</a>>>
wrote:<br>
><br>
> yes.. in case of GNR (GPFS native raid) .. we do
end-to-end<br>
> check-summing ... client --> server -->
downToDisk<br>
> GNR writes down a chksum to disk (to all pdisks /all
"raid" segments<br>
> ) so that dropped writes can be detected as well as
miss-done<br>
> writes (bit flips..)<br>
><br>
><br>
><br>
> From: Aaron Knister <<a
href="mailto:aaron.s.knister@nasa.gov" target="_blank"
moz-do-not-send="true">aaron.s.knister@nasa.gov</a><br>
> <mailto:<a href="mailto:aaron.s.knister@nasa.gov"
target="_blank" moz-do-not-send="true">aaron.s.knister@nasa.gov</a>>><br>
> To: gpfsug main discussion list <<a
href="mailto:gpfsug-discuss@spectrumscale.org"
target="_blank" moz-do-not-send="true">gpfsug-discuss@spectrumscale.org</a><br>
> <mailto:<a
href="mailto:gpfsug-discuss@spectrumscale.org"
target="_blank" moz-do-not-send="true">gpfsug-discuss@spectrumscale.org</a>>><br>
> Date: 06/30/2017 07:15 PM<br>
> Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum
Scale (GPFS):<br>
> RDMA-enabled network adapter failure on the NSD
server may result in<br>
> file IO error (2017.06.30)<br>
> Sent by: <a
href="mailto:gpfsug-discuss-bounces@spectrumscale.org"
target="_blank" moz-do-not-send="true">gpfsug-discuss-bounces@spectrumscale.org</a><br>
> <mailto:<a
href="mailto:gpfsug-discuss-bounces@spectrumscale.org"
target="_blank" moz-do-not-send="true">gpfsug-discuss-bounces@spectrumscale.org</a>><br>
>
------------------------------------------------------------------------<br>
><br>
><br>
><br>
> I'm curious to know why this doesn't affect GSS/ESS?
Is it a feature of<br>
> the additional check-summing done on those platforms?<br>
><br>
><br>
> -------- Forwarded Message --------<br>
> Subject: FLASH: IBM Spectrum Scale (GPFS):
RDMA-enabled network<br>
> adapter<br>
> failure on the NSD server may result in file IO error
(2017.06.30)<br>
> Date: Fri, 30 Jun 2017 14:19:02
+0000<br>
> From: IBM My Notifications<br>
> <<a href="mailto:mynotify@stg.events.ihost.com"
target="_blank" moz-do-not-send="true">mynotify@stg.events.ihost.com</a>
<mailto:<a href="mailto:mynotify@stg.events.ihost.com"
target="_blank" moz-do-not-send="true">mynotify@stg.events.ihost.com</a>>><br>
> To: <a href="mailto:aaron.s.knister@nasa.gov"
target="_blank" moz-do-not-send="true">aaron.s.knister@nasa.gov</a>
<mailto:<a href="mailto:aaron.s.knister@nasa.gov"
target="_blank" moz-do-not-send="true">aaron.s.knister@nasa.gov</a>><br>
><br>
><br>
><br>
><br>
> My Notifications for Storage - 30 Jun 2017<br>
><br>
> Dear Subscriber (<a
href="mailto:aaron.s.knister@nasa.gov" target="_blank"
moz-do-not-send="true">aaron.s.knister@nasa.gov</a><br>
> <mailto:<a href="mailto:aaron.s.knister@nasa.gov"
target="_blank" moz-do-not-send="true">aaron.s.knister@nasa.gov</a>>),<br>
><br>
> Here are your updates from IBM My Notifications.<br>
><br>
> Your support Notifications display in English by
default. Machine<br>
> translation based on your IBM profile<br>
> language setting is added if you specify this option
in My defaults<br>
> within My Notifications.<br>
> (Note: Not all languages are available at this time,
and the English<br>
> version always takes precedence<br>
> over the machine translated version.)<br>
><br>
>
------------------------------------------------------------------------------<br>
> 1. IBM Spectrum Scale<br>
><br>
> - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled
network adapter<br>
> failure<br>
> on the NSD server may result in file IO error<br>
> - URL:<br>
> <a
href="http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E</a><br>
> <<a
href="http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E</a>><br>
> - ABSTRACT: IBM has identified an issue with all IBM
GPFS and IBM<br>
> Spectrum Scale versions where the NSD server is
enabled to use RDMA for<br>
> file IO and the storage used in your GPFS cluster
accessed via NSD<br>
> servers (not fully SAN accessible) includes anything
other than IBM<br>
> Elastic Storage Server (ESS) or GPFS Storage Server
(GSS); under these<br>
> conditions, when the RDMA-enabled network adapter
fails, the issue may<br>
> result in undetected data corruption for file write
or read operations.<br>
><br>
>
------------------------------------------------------------------------------<br>
> Manage your My Notifications subscriptions, or send
questions and<br>
> comments.<br>
> - Subscribe or Unsubscribe -<br>
> <a href="https://www.ibm.com/support/mynotifications"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.ibm.com/support/mynotifications</a><br>
> <<a
href="https://www.ibm.com/support/mynotifications"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.ibm.com/support/mynotifications</a>><br>
> - Feedback -<br>
> <a
href="https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html</a><br>
> <<a
href="https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html</a>><br>
><br>
> - Follow us on Twitter - <a
href="https://twitter.com/IBMStorageSupt" rel="noreferrer"
target="_blank" moz-do-not-send="true">https://twitter.com/IBMStorageSupt</a><br>
> <<a href="https://twitter.com/IBMStorageSupt"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://twitter.com/IBMStorageSupt</a>><br>
><br>
><br>
><br>
> To ensure proper delivery please add <a
href="mailto:mynotify@stg.events.ihost.com" target="_blank"
moz-do-not-send="true">mynotify@stg.events.ihost.com</a><br>
> <mailto:<a
href="mailto:mynotify@stg.events.ihost.com" target="_blank"
moz-do-not-send="true">mynotify@stg.events.ihost.com</a>>
to<br>
> your address book.<br>
> You received this email because you are subscribed to
IBM My<br>
> Notifications as:<br>
> <a href="mailto:aaron.s.knister@nasa.gov"
target="_blank" moz-do-not-send="true">aaron.s.knister@nasa.gov</a>
<mailto:<a href="mailto:aaron.s.knister@nasa.gov"
target="_blank" moz-do-not-send="true">aaron.s.knister@nasa.gov</a>><br>
><br>
> Please do not reply to this message as it is
generated by an automated<br>
> service machine.<br>
><br>
> (C) International Business Machines Corporation 2017.
All rights<br>
> reserved.<br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">spectrumscale.org</a>
<<a href="http://spectrumscale.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://spectrumscale.org</a>><br>
> <a
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
> <<a
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>><br>
><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">spectrumscale.org</a>
<<a href="http://spectrumscale.org" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://spectrumscale.org</a>><br>
> <a
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
> <<a
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at <a href="http://spectrumscale.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">spectrumscale.org</a><br>
> <a
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
><br>
<br>
--<br>
Aaron Knister<br>
NASA Center for Climate Simulation (Code 606.2)<br>
Goddard Space Flight Center<br>
<a href="tel:%28301%29%20286-2776" value="+13012862776"
target="_blank" moz-do-not-send="true">(301) 286-2776</a><br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org"
rel="noreferrer" target="_blank" moz-do-not-send="true">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
<a class="moz-txt-link-freetext" href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a>
</pre>
</blockquote>
<br>
</body>
</html>