[gpfsug-discuss] Alternative to Scale S3 API.

Wed Oct 28 06:45:29 GMT 2020

Hi Andi

The s3api middleware does support multipart uploads..

https://docs.openstack.org/swift/latest/s3_compat.html

The current version of Swift (PIKE) that is bundled with Spectrum Scale
5.0.X doesn't..
https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.5/com.ibm.spectrum.scale.v5r05.doc/bl1adm_ManagingOpenStackACLsviaAmazonS3API.htm

According to the Spectrum Scale Roadmap, 5.1 is due out 2H20.. Not sure if
someone from development can confirm the GA date.

Does Veeam have a test utility? You could always test it using the current
Swift AIO or if you can provide me with a test utility I can test that for
you.

Kind Regards

Nishaan Docrat
Client Technical Specialist - Storage Systems
IBM Systems Hardware

Work: +27 (0)11 302 5001
Mobile: +27 (0)81 040 3793
Email: nishaan at za.ibm.com

From:	Andi Christiansen <andi at christiansen.xxx>
To:	gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>,
            Nishaan Docrat <NISHAAN at za.ibm.com>
Date:	2020/10/28 08:24
Subject:	[EXTERNAL] Re: [gpfsug-discuss] Alternative to Scale S3 API.

Hi Nishaan, Thanks for you reply. When you say 5.1? is that 5.1.x.x or
5.0.5.1? Some...

                   This Message Is From an External Sender                  
                   This message came from outside your organization.        

Hi Nishaan,

Thanks for you reply.

When you say 5.1? is that 5.1.x.x or 5.0.5.1?

Some of the limitations we have encountered is the multipart upload not
being supported and some md5sum that the s3 api does that veeam actually
dont like. also interms of the management on the Scale GUI, that has to be
on one of the S3 CES nodes in order to be able to show project, container
etc... but when you have a HAProxy for enabling SSL then a GUI is not
available as they both use port 443?

i know min.io is not the full stack of S3 API commands but as far as i can
read it comes with more features out of the box than Scale S3 does,
multipart for an example...

I looked through your documentation and its very close to what we have set
up today and found to not work...

If multipart uploads would be supported today on scale S3 i would think
about still using scale for the s3 part but as i expect that you talk about
5.1.x.x i dont see that being released any time soon? and dont know if that
is actually going to be supported in that release then i cant wait for that
to happen..

Best Regards
Andi Christiansen
      On 10/27/2020 2:38 PM Nishaan Docrat <nishaan at za.ibm.com> wrote:

      Hi Andi

      The current S3 compatibility in Spectrum Scale is delivered via the
      Swift3 middleware. This middleware has since been replaced by s3api
      in later versions of Swift. Spectrum Scale 5.1 will make use of Swift
      Train release which will include the new s3api middleware.

      I've tested the S3 compatibility with a few applications including
      Spectrum Scale itself (i.e. Cloud Data Sharing to another Scale
      Object store using S3 API) and Spectrum Protect etc. and haven't had
      any issue. I've also ran a few application tools to test for an S3
      compliant object stores and again had no issues.

      You can use s3compat to test the current compatibility.. Or you can
      check here for the current compatibility..
      https://docs.openstack.org/swift/latest/s3_compat.html

      Not sure if there is any other way to talk HTTPS without using
      HAProxy.

      In any case, I've documented the process to setup an S3 compliant
      object store including supporting virtual-hosted style bucket
      addressing which you can find here..

      https://www.linkedin.com/feed/update/urn:li:activity:6720227398756909056/

      Most storage vendors including minio would not support the full S3
      API stack as alot of the calls are specific to AWS (like billing
      stuff etc.).

      Anyway, good luck with your testing.

      Kind Regards

      Nishaan Docrat
      Client Technical Specialist - Storage Systems
      IBM Systems Hardware

      Work: +27 (0)11 302 5001
      Mobile: +27 (0)81 040 3793
      Email: nishaan at za.ibm.com

      Inactive hide details for Andi Christiansen ---2020/10/27
      13:59:30---Hi all, We have over a longer period used the S3 API withi
      Andi Christiansen ---2020/10/27 13:59:30---Hi all, We have over a
      longer period used the S3 API within spectrum Scale.. And that has
      shown that

      From: Andi Christiansen <andi at christiansen.xxx>
      To: "gpfsug-discuss at spectrumscale.org"
      <gpfsug-discuss at spectrumscale.org>
      Date: 2020/10/27 13:59
      Subject: [EXTERNAL] [gpfsug-discuss] Alternative to Scale S3 API.
      Sent by: gpfsug-discuss-bounces at spectrumscale.org

      Hi all,

      We have over a longer period used the S3 API within spectrum Scale..
      And that has shown that it does not support very many applications
      because of limitations of the API..

      Has anyone got any experience with any other product we can deploy
      on-top of Spectrum Scale that will give us a true S3 API with full
      functionalities and able to answer on port 443? As of now we use
      HAProxy to forware ssl request back and forth from Scale S3 API.

      We have looked at MinIO which seems to be fairly simple and maybe
      might solve a lot of incompatibilities with clients software. But the
      product seems to be very badly documented at least for me.

      The idea is basically that a client uses their backup application
      (rubrik, veeam etc.) to connect to a domain(for example
      backup.mycompany.com) with their access and secret key and have
      access to their bucket only. and it must be over https/ssl.

      If someone has any knowledge to minio or any other product that might
      solve our problem I will be glad to hear from you! 😊

      Thank you in advance!
      _______________________________________________
      gpfsug-discuss mailing list
      gpfsug-discuss at spectrumscale.org
      http://gpfsug.org/mailman/listinfo/gpfsug-discuss

      _______________________________________________
      gpfsug-discuss mailing list
      gpfsug-discuss at spectrumscale.org
      http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20201028/4c5ddab4/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 19991351.jpg
Type: image/jpeg
Size: 35643 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20201028/4c5ddab4/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20201028/4c5ddab4/attachment-0002.gif>