[gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter failure on the NSD server may result in file IO error (2017.06.30)

Aaron Knister aaron.s.knister at nasa.gov
Tue Aug 22 17:45:00 BST 2017


(I'm slowly catching up on a backlog of e-mail, sorry for the delayed 
reply).

Thanks, Sven. I recognize the complexity and appreciate your explanation.

In my mind I had envisioned either the block integrity information being 
stored as a new metadata structure or stored leveraging T10-DIX/DIF 
(perhaps configurable on a per-pool basis) to pass the checksums down to 
the RAID controller.

I would quite like to run GNR as software on generic hardware and in 
fact voted, along with 26 other customers, on an RFE 
(https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090) 
requesting this but the request was declined. I think customers spoke 
pretty loudly there and IBM gave it the kibosh.

-Aaron

On 06/30/2017 02:25 PM, Sven Oehme wrote:
>
> end-to-end data integrity is very important and the reason it hasn't 
> been done in Scale is not because its not important, its because its 
> very hard to do without impacting performance in a very dramatic way.
>
> imagine your raid controller blocksize is 1mb and your filesystem 
> blocksize is 1MB . if your application does a 1 MB write this ends up 
> being a perfect full block , full track de-stage to your raid layer 
> and everything works fine and fast. as soon as you add checksum 
> support you need to add data somehow into this, means your 1MB is no 
> longer 1 MB but 1 MB+checksum.
>
> to store this additional data you have multiple options, inline , 
> outside the data block or some combination ,the net is either you need 
> to do more physical i/o's to different places to get both the data and 
> the corresponding checksum or your per block on disc structure becomes 
> bigger than than what your application reads/or writes, both put 
> massive burden on the Storage layer as e.g. a 1 MB write will now, 
> even the blocks are all aligned from the application down to the raid 
> layer, cause a read/modify/write on the raid layer as the data is 
> bigger than the physical track size.
>
> so to get end-to-end checksum in Scale outside of ESS the best way is 
> to get GNR as SW to run on generic HW, this is what people should vote 
> for as RFE if they need that functionality. beside end-to-end 
> checksums you get read/write cache and acceleration , fast rebuild and 
> many other goodies as a added bonus.
>
> Sven
>
>
> On Fri, Jun 30, 2017 at 10:53 AM Aaron Knister 
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>
>     In fact the answer was quite literally "no":
>
>     https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523
>     (the RFE was declined and the answer was that the "function is already
>     available in GNR environments").
>
>     Regarding GNR, see this RFE request
>     https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090
>     requesting the use of GNR outside of an ESS/GSS environment. It's
>     interesting to note this is the highest voted Public RFE for GPFS
>     that I
>     can see, at least. It too was declined.
>
>     -Aaron
>
>     On 6/30/17 1:41 PM, Aaron Knister wrote:
>     > Thanks Olaf, that's good to know (and is kind of what I
>     suspected). I've
>     > requested a number of times this capability for those of us who
>     can't
>     > use or aren't using GNR and the answer is effectively "no". This
>     > response is curious to me because I'm sure IBM doesn't believe
>     that data
>     > integrity is only important and of value to customers who
>     purchase their
>     > hardware *and* software.
>     >
>     > -Aaron
>     >
>     > On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser
>     <olaf.weiser at de.ibm.com <mailto:olaf.weiser at de.ibm.com>
>     > <mailto:olaf.weiser at de.ibm.com <mailto:olaf.weiser at de.ibm.com>>>
>     wrote:
>     >
>     >     yes.. in case of GNR (GPFS native raid) .. we do end-to-end
>     >     check-summing ...  client --> server --> downToDisk
>     >     GNR writes down a chksum to disk (to all pdisks /all "raid"
>     segments
>     >     )  so that dropped writes can be detected as well as miss-done
>     >     writes (bit flips..)
>     >
>     >
>     >
>     >     From: Aaron Knister <aaron.s.knister at nasa.gov
>     <mailto:aaron.s.knister at nasa.gov>
>     >     <mailto:aaron.s.knister at nasa.gov
>     <mailto:aaron.s.knister at nasa.gov>>>
>     >     To: gpfsug main discussion list
>     <gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>
>     >     <mailto:gpfsug-discuss at spectrumscale.org
>     <mailto:gpfsug-discuss at spectrumscale.org>>>
>     >     Date: 06/30/2017 07:15 PM
>     >     Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS):
>     >     RDMA-enabled network adapter failure on the NSD server may
>     result in
>     >     file IO error (2017.06.30)
>     >     Sent by: gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>     >     <mailto:gpfsug-discuss-bounces at spectrumscale.org
>     <mailto:gpfsug-discuss-bounces at spectrumscale.org>>
>     >
>      ------------------------------------------------------------------------
>     >
>     >
>     >
>     >     I'm curious to know why this doesn't affect GSS/ESS? Is it a
>     feature of
>     >     the additional check-summing done on those platforms?
>     >
>     >
>     >     -------- Forwarded Message --------
>     >     Subject:  FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network
>     >     adapter
>     >     failure on the NSD server may result in file IO error
>     (2017.06.30)
>     >     Date:                  Fri, 30 Jun 2017 14:19:02 +0000
>     >     From:                  IBM My Notifications
>     >     <mynotify at stg.events.ihost.com
>     <mailto:mynotify at stg.events.ihost.com>
>     <mailto:mynotify at stg.events.ihost.com
>     <mailto:mynotify at stg.events.ihost.com>>>
>     >     To: aaron.s.knister at nasa.gov
>     <mailto:aaron.s.knister at nasa.gov> <mailto:aaron.s.knister at nasa.gov
>     <mailto:aaron.s.knister at nasa.gov>>
>     >
>     >
>     >
>     >
>     >     My Notifications for Storage - 30 Jun 2017
>     >
>     >     Dear Subscriber (aaron.s.knister at nasa.gov
>     <mailto:aaron.s.knister at nasa.gov>
>     >     <mailto:aaron.s.knister at nasa.gov
>     <mailto:aaron.s.knister at nasa.gov>>),
>     >
>     >     Here are your updates from IBM My Notifications.
>     >
>     >     Your support Notifications display in English by default.
>     Machine
>     >     translation based on your IBM profile
>     >     language setting is added if you specify this option in My
>     defaults
>     >     within My Notifications.
>     >     (Note: Not all languages are available at this time, and the
>     English
>     >     version always takes precedence
>     >     over the machine translated version.)
>     >
>     >
>      ------------------------------------------------------------------------------
>     >     1. IBM Spectrum Scale
>     >
>     >     - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter
>     >     failure
>     >     on the NSD server may result in file IO error
>     >     - URL:
>     >
>     http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E
>     >   
>      <http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E>
>     >     - ABSTRACT: IBM has identified an issue with all IBM GPFS
>     and IBM
>     >     Spectrum Scale versions where the NSD server is enabled to
>     use RDMA for
>     >     file IO and the storage used in your GPFS cluster accessed
>     via NSD
>     >     servers (not fully SAN accessible) includes anything other
>     than IBM
>     >     Elastic Storage Server (ESS) or GPFS Storage Server (GSS);
>     under these
>     >     conditions, when the RDMA-enabled network adapter fails, the
>     issue may
>     >     result in undetected data corruption for file write or read
>     operations.
>     >
>     >
>      ------------------------------------------------------------------------------
>     >     Manage your My Notifications subscriptions, or send
>     questions and
>     >     comments.
>     >     - Subscribe or Unsubscribe -
>     > https://www.ibm.com/support/mynotifications
>     >     <https://www.ibm.com/support/mynotifications>
>     >     - Feedback -
>     >
>     https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html
>     >   
>      <https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html>
>     >
>     >     - Follow us on Twitter - https://twitter.com/IBMStorageSupt
>     >     <https://twitter.com/IBMStorageSupt>
>     >
>     >
>     >
>     >     To ensure proper delivery please add
>     mynotify at stg.events.ihost.com <mailto:mynotify at stg.events.ihost.com>
>     >     <mailto:mynotify at stg.events.ihost.com
>     <mailto:mynotify at stg.events.ihost.com>> to
>     >     your address book.
>     >     You received this email because you are subscribed to IBM My
>     >     Notifications as:
>     > aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>
>     <mailto:aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>>
>     >
>     >     Please do not reply to this message as it is generated by an
>     automated
>     >     service machine.
>     >
>     >     (C) International Business Machines Corporation 2017. All rights
>     >     reserved.
>     >     _______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at spectrumscale.org
>     <http://spectrumscale.org> <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
>     >
>     >
>     >
>     >
>     >     _______________________________________________
>     >     gpfsug-discuss mailing list
>     >     gpfsug-discuss at spectrumscale.org
>     <http://spectrumscale.org> <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     >     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > gpfsug-discuss mailing list
>     > gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     >
>
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776 <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20170822/d3a0aecb/attachment-0001.htm>


More information about the gpfsug-discuss mailing list