<font size=2 face="sans-serif">Hi,</font>
<br>
<br><font size=2 face="sans-serif">there are various ways to speed up backup
of data from GPFS to TSM (which btw is the same problem for most other
backup solutions as well), but first one needs to find out what the problem
is as it can be multiple completely independent issues and each of them
needs a different solution, so let me explain the different issues and
also what you can do about them.</font>
<br>
<br><font size=2 face="sans-serif">the very first challenge is to find
what data has changed. the way TSM does this is by crawling trough your
filesystem, looking at mtime on each file to find out which file has changed.
think about a ls -Rl on your filesystem root. this can, depending on how
many files you have, take days in a large scale environment (think 100's
of millions of files). there is very little one can speed up on this
process the way its done. all you can do is put metadata on faster disks
(e.g. SSD) and that will improve the speed of this 'scan phase'. an alternative
is to not do this scan at all with the TSM client, but instead let GPFS
find out for TSM what files have changed and then share this information
with TSM . the function/command in GPFS to do so is called mmbackup : </font>
<br>
<br><a href="https://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r5.gpfs100.doc%2Fbl1adm_backupusingmmbackup.htm"><font size=2 color=blue face="sans-serif">https://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r5.gpfs100.doc%2Fbl1adm_backupusingmmbackup.htm</font></a>
<br>
<br><font size=2 face="sans-serif">it essentially traverses the GPFS metadata
sequentially and in parallel across all nodes and filters out files that
needs to be backed up. </font>
<br><font size=2 face="sans-serif">in several customer environments i was
called to assist with issues like this, this change alone did speed up
the backup process multiple orders of magnitudes. we had a few customers
where this change reduced the scan time from days down to minutes. its
not always this big, but its usually the largest chunk of the issue. </font>
<br>
<br><font size=2 face="sans-serif">the 2nd challenge is if you have to
backup a very large number (millions) of very small (<32k) files. </font>
<br><font size=2 face="sans-serif">the main issue here is that for each
file TSM issues a random i/o to GPFS, one at a time, so your throughput
directly correlates with size of the files and latency for a single file
read operation. if you are not on 3.5 TL3 and/or your files don't fit into
the inode its actually even 2 random i/os that are issued as you need to
read the metadata followed by the data block for the file. </font>
<br><font size=2 face="sans-serif">in this scenario you can only do 2 things
: </font>
<br>
<br><font size=2 face="sans-serif">1. parallelism - mmbackup again starts
multiple processes in parallel to speed up this phase of the backup </font>
<br><font size=2 face="sans-serif">2. use a 'helper' process to prefetch
data for a single TSM client so all data comes out of cache and the latency
for the random reads is eliminated to increase throughput.</font>
<br>
<br><font size=2 face="sans-serif">without any of this seeing only a few
low MB/sec is not uncommon for customers, but with the changes above you
are able to backup very large quantities of data. </font>
<br>
<br><font size=2 face="sans-serif">hope this helps. Sven</font>
<br>
<br><font size=2 face="sans-serif">------------------------------------------<br>
Sven Oehme <br>
Scalable Storage Research <br>
email: oehmes@us.ibm.com <br>
IBM Almaden Research Lab <br>
------------------------------------------</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Sabuj Pattanayek <sabujp@gmail.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug main discussion
list <gpfsug-discuss@gpfsug.org></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">03/20/2014 06:39 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@gpfsug.org</font>
<br>
<hr noshade>
<br>
<br>
<br><font size=3>We're using tsm 7.1 with gpfs 3.5.0.11. At some point
we do want to enable the HSM features but haven't had time to properly
configure/set them up yet. I had dmapi enabled on GPFS but was never able
to bring it up with dmapi enabled. Everything wasn't properly configured
at the time and we were missing some pieces (not my post, but same issue)
:</font>
<br>
<br><a href="https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591"><font size=3 color=blue><u>https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014622591</u></font></a>
<br>
<br><font size=3>I'd say that we are having less than optimal performance
with TSM however. We're only able to pull about 4TB a day. It took us 20
days to backup 80TB for our initial full. Using rsync/tar piped to tar
would probably have taken less than a week. We tried various methods, e.g.
using a fast intermediate diskpool, going simultaneously to our 6 LTO6
tape drives, etc but each "cp" (tsm client) process that TSM
would use seemed to be very slow. We tweaked just about every setting to
optimize performance but to really no avail. When going to the disk pool
this is what should have happened :</font>
<br>
<br><font size=3>GPFS => relatively fast random I/O (on par with rsync/tar
piped to tar) tsm disk cache => large sequential I/O's for each disk
pool volume => tape</font>
<br>
<br><font size=3>this is what really happened</font>
<br>
<br><font size=3>GPFS => slow random I/O => tsm disk pool cache =>
slow random I/O => tape</font>
<br>
<br><font size=3>so instead we did :</font>
<br>
<br><font size=3>GPFS => slow random I/O (TSM) => tape</font>
<br>
<br><font size=3>..but was the same speed as going through the tsm disk
pool cache. We closely monitored the network, disk, memory, cpu, on the
tsm server and none of the hardware or capabilities of the server were
the bottleneck, it was all in TSM.</font>
<br>
<br><font size=3>If anyone has seen this sort of behavior and has some
pointers/hints at improving performance I'd be glad to hear it.</font>
<br>
<br><font size=3>Thanks,</font>
<br><font size=3>Sabuj</font>
<br><font size=3><br>
</font>
<br><font size=3>On Thu, Mar 20, 2014 at 5:21 PM, Grace Tsai <</font><a href=mailto:gtsai@slac.stanford.edu target=_blank><font size=3 color=blue><u>gtsai@slac.stanford.edu</u></font></a><font size=3>>
wrote:</font>
<br><font size=3>Hi,<br>
<br>
Is TSM/HSM 7.1 compatible with GPFS 3.5.0.12 ?<br>
<br>
Thanks.<br>
<br>
Grace<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at </font><a href=http://gpfsug.org/ target=_blank><font size=3 color=blue><u>gpfsug.org</u></font></a><font size=3 color=blue><u><br>
</u></font><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target=_blank><font size=3 color=blue><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></font></a>
<br><tt><font size=2>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>
</font></tt>
<br>