[gpfsug-discuss] Using AFM to migrate files. (Bill Pappas to Loric Totay

Bill Pappas bpappas at dstonline.com
Mon Oct 24 19:03:07 BST 2016


>>For the record, I have completed today my AFM migration of a filesystem with 100 million files. Users are now accessing the new filesystem.


Loric-> Hi.  I was wondering, what version of GPFS where you running on the home and cache clusters?  I take it you broke up the prefetch list into smaller (for example <2 million) file lists? If not, how? How much capacity did you migfrate over and how long did this process take? Thanks.


Bill Pappas

901-619-0585

bpappas at dstonline.com


[1466780990050_DSTlogo.png]


[http://www.prweb.com/releases/2016/06/prweb13504050.htm]

http://www.prweb.com/releases/2016/06/prweb13504050.htm


________________________________
From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> on behalf of gpfsug-discuss-request at spectrumscale.org <gpfsug-discuss-request at spectrumscale.org>
Sent: Monday, October 24, 2016 12:05 PM
To: gpfsug-discuss at spectrumscale.org
Subject: gpfsug-discuss Digest, Vol 57, Issue 60

Send gpfsug-discuss mailing list submissions to
        gpfsug-discuss at spectrumscale.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
        gpfsug-discuss-request at spectrumscale.org

You can reach the person managing the list at
        gpfsug-discuss-owner at spectrumscale.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. Using AFM to migrate within the same cluster (Eric Horst)
   2. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs)
      - URL encoding for pathnames (Loic Tortay)
   3. Re: Using AFM to migrate within the same cluster (Scott Fadden)
   4. Re: Using AFM to migrate files. (Peter Childs) (Peter Childs)
      - URL encoding for pathnames (Yaron Daniel)


----------------------------------------------------------------------

Message: 1
Date: Mon, 24 Oct 2016 09:16:56 -0700
From: Eric Horst <erich at uw.edu>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: [gpfsug-discuss] Using AFM to migrate within the same cluster
Message-ID:
        <CAF-To-2_BQSXKY1Qv2gs6BnxOC2RU=ywfiL9McW5CRehZvDpiw at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

The recent conversation about AFM has been interesting. I've read the
documentation several times and this is my question. Can AFM be used
to migrate between two filesystems in the same cluster? There are
examples of moving between clusters with NFS or native protocol but
I've got a simple situation of needing to transparently move 100M
files between two existing filesystems.

Thanks,

-Eric

--
Eric Horst
University of Washington


------------------------------

Message: 2
Date: Mon, 24 Oct 2016 18:50:51 +0200
From: Loic Tortay <tortay at cc.in2p3.fr>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter
        Childs) (Peter Childs) - URL encoding for pathnames
Message-ID: <f6d09494-e9c9-cb08-3ab5-5d26371aacbc at cc.in2p3.fr>
Content-Type: text/plain; charset=utf-8

On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote:
>
> mmafmctl prefecth expects encoded list file, and it is not documented
> correctly.  Issues like memory leak, file descriptor leak, and fileset
> going into Unmounted state were fixed in later releases (4.2.1/4.2.2). All
> your points are correct with respect to AFM migration. There is manual
> intervention required. Also prefetch does not give list of files which
> were failed during data read. Users need to run policy to find all
> uncached files today.
>
Hello,
For the record, I have completed today my AFM migration of a filesystem
with 100 million files. Users are now accessing the new filesystem.

After disabling user access and a last "prefetch", the AFM filesets were
converted to independent filesets.
Less than 600 files were then found to be different between the "home"
and the "cache" filesystems with a metadata comparison (I just copied
the files from the old filesystem to the new one).
I have compared the MD5 of a few thousand randomly selected files and
found no differences between the "home" and the "cache" filesystems.
I expect the users to let us know if they find something different (they
have been instructed to do so). We'll keep the "home" filesystem around
for some time, just in case there is a problem.

Maybe something else that should be mentionned in the documentation is
what to do with the ".ptrash" directories after the AFM filesets have
been converted. I removed them since they contained files that had
clearly been deleted by the users.


Lo?c.
--
|       Lo?c Tortay <tortay at cc.in2p3.fr> - IN2P3 Computing Centre      |


------------------------------

Message: 3
Date: Mon, 24 Oct 2016 09:57:33 -0700
From: "Scott Fadden" <sfadden at us.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Using AFM to migrate within the same
        cluster
Message-ID:
        <OF42D09886.A30A7C7B-ON88258056.005D0734-88258056.005D2900 at notes.na.collabserv.com>

Content-Type: text/plain; charset="us-ascii"

Yes you can use AFM to move data within a cluster. If you are using the
NSD protocol the target needs to be a separate file system, if you are
using NFS it needs to be an NFS export.


Scott Fadden
Spectrum Scale - Technical Marketing
Phone: (503) 880-5833
sfadden at us.ibm.com
http://www.ibm.com/systems/storage/spectrum/scale



From:   Eric Horst <erich at uw.edu>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   10/24/2016 09:17 AM
Subject:        [gpfsug-discuss] Using AFM to migrate within the same
cluster
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



The recent conversation about AFM has been interesting. I've read the
documentation several times and this is my question. Can AFM be used
to migrate between two filesystems in the same cluster? There are
examples of moving between clusters with NFS or native protocol but
I've got a simple situation of needing to transparently move 100M
files between two existing filesystems.

Thanks,

-Eric

--
Eric Horst
University of Washington
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/866c55cf/attachment-0001.html>

------------------------------

Message: 4
Date: Mon, 24 Oct 2016 20:05:00 +0300
From: "Yaron Daniel" <YARD at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Using AFM to migrate files. (Peter
        Childs) (Peter Childs) - URL encoding for pathnames
Message-ID:
        <OFBC94A30E.47D6D231-ONC2258056.005DCCB6-C2258056.005DD821 at notes.na.collabserv.com>

Content-Type: text/plain; charset="iso-8859-1"

Hi

Maybe worth also to check if there are any orphan files in the NEW fs ?


Regards





Yaron Daniel
 94 Em Ha'Moshavot Rd

Server, Storage and Data Services - Team Leader
 Petach Tiqva, 49527
Global Technology Services
 Israel
Phone:
+972-3-916-5672


Fax:
+972-3-916-5672


Mobile:
+972-52-8395593


e-mail:
yard at il.ibm.com


IBM Israel









From:   Loic Tortay <tortay at cc.in2p3.fr>
To:     gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date:   10/24/2016 07:50 PM
Subject:        Re: [gpfsug-discuss] Using AFM to migrate files. (Peter
Childs) (Peter Childs) - URL encoding for pathnames
Sent by:        gpfsug-discuss-bounces at spectrumscale.org



On 10/24/2016 11:44 AM, Venkateswara R Puvvada wrote:
>
> mmafmctl prefecth expects encoded list file, and it is not documented
> correctly.  Issues like memory leak, file descriptor leak, and fileset
> going into Unmounted state were fixed in later releases (4.2.1/4.2.2).
All
> your points are correct with respect to AFM migration. There is manual
> intervention required. Also prefetch does not give list of files which
> were failed during data read. Users need to run policy to find all
> uncached files today.
>
Hello,
For the record, I have completed today my AFM migration of a filesystem
with 100 million files. Users are now accessing the new filesystem.

After disabling user access and a last "prefetch", the AFM filesets were
converted to independent filesets.
Less than 600 files were then found to be different between the "home"
and the "cache" filesystems with a metadata comparison (I just copied
the files from the old filesystem to the new one).
I have compared the MD5 of a few thousand randomly selected files and
found no differences between the "home" and the "cache" filesystems.
I expect the users to let us know if they find something different (they
have been instructed to do so). We'll keep the "home" filesystem around
for some time, just in case there is a problem.

Maybe something else that should be mentionned in the documentation is
what to do with the ".ptrash" directories after the AFM filesets have
been converted. I removed them since they contained files that had
clearly been deleted by the users.


Lo?c.
--
|       Lo?c Tortay <tortay at cc.in2p3.fr> - IN2P3 Computing Centre      |
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20161024/449a0cf1/attachment.gif>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 57, Issue 60
**********************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20161024/fff3f370/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OutlookEmoji-1466780990050_DSTlogo.png.png
Type: image/png
Size: 6282 bytes
Desc: OutlookEmoji-1466780990050_DSTlogo.png.png
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20161024/fff3f370/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg
Type: image/jpeg
Size: 14887 bytes
Desc: OutlookEmoji-http://www.prweb.com/releases/2016/06/prweb13504050.htm.jpg
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20161024/fff3f370/attachment-0001.jpg>


More information about the gpfsug-discuss mailing list