<font size=2 face="sans-serif">Hi Alex, Stewart,</font>
<br>
<br><font size=2 face="sans-serif">the problem is more likely a latency
and lack of parallelism of the transfer.</font>
<br><font size=2 face="sans-serif"> </font>
<br><font size=2 face="sans-serif">if you have 1 thread that transfers
files of 1k in size you don't get a lot of bandwidth as it transfers one
at a time and the latency kills the bandwidth. to explain this on an example
:</font>
<br>
<br><font size=2 face="sans-serif">assume your network between client and
server is 10 gigabit and both nodes are capable of pushing this.</font>
<br>
<br><font size=2 face="sans-serif">if it takes 1 ms for reading + writing
each 1 kb file you get ~ 1.02 MB/sec</font>
<br><font size=2 face="sans-serif">if your filesize changes to 4k, even
you don't do anything else it goes up to 4.06 MB/sec</font>
<br><font size=2 face="sans-serif">if you can reduce latency on read or
write to lets say 100us the same process would transfer 37.86 MB/sec</font>
<br>
<br><font size=2 face="sans-serif">so this is a latency / parallelism problem
and this has nothing to do with GPFS, you would have exactly the same issue.</font>
<br><font size=2 face="sans-serif">the solution is to copy the data with
a tool that does it multi threaded as the numbers above are based on 1
thread. </font>
<br><font size=2 face="sans-serif">if you would have 100us read+write time
and transfer 4k files with 10 threads in parallel the same transfer would
be close to 400 MB/sec.</font>
<br>
<br><font size=2 face="sans-serif">hope that helps. Sven </font>
<br><font size=2 face="sans-serif">------------------------------------------<br>
Sven Oehme <br>
Scalable Storage Research <br>
email: oehmes@us.ibm.com <br>
Phone: +1 (408) 824-8904 <br>
IBM Almaden Research Lab <br>
------------------------------------------</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Alex Chekholko <chekh@stanford.edu></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gpfsug-discuss@gpfsug.org</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">05/21/2014 01:05 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [gpfsug-discuss]
Mitigating Poor Small-file I/O Performance</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gpfsug-discuss-bounces@gpfsug.org</font>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>Hi Stewart,<br>
<br>
First, a good simple reproducible benchmark is fdtree:<br>
</font></tt><a href="https://computing.llnl.gov/?set=code&page=sio_downloads"><tt><font size=2>https://computing.llnl.gov/?set=code&page=sio_downloads</font></tt></a><tt><font size=2><br>
<br>
Something simple like this should take a min or two:<br>
bash fdtree.bash -l 3 -s 64<br>
<br>
Or the same exact small run can take up to hours on a slow system.<br>
<br>
For GPFS, since it's a clustered filesystem, first you need to make sure
<br>
you're looking at the aggregate performance and not just on one client.
<br>
Perhaps your filesystem is performing great, but it's maxed out
at <br>
that moment when you run your test from your single client. So you
need <br>
to be able to monitor the disk system.<br>
<br>
In general, the answer to your question is, in order of simplicity: add
<br>
more spindles, possibly also separate the metadata out to separate <br>
storage, possibly make your filesystem block size smaller.<br>
<br>
The first you can do by adding more hardware, the second is easier when
<br>
you design your whole system, though possible to do on a running <br>
filesystem. The third can only be done at filesystem creation.<br>
<br>
For "small files", how "small" is "small".
I guess generally we mean <br>
smaller than filesystem block size.<br>
<br>
Regards,<br>
Alex<br>
<br>
<br>
On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:<br>
> Hi All,<br>
><br>
> My name is Stewart Howard and I work for Indiana University as an
admin<br>
> on a two-site replicated GPFS cluster. I'm a new member of this
mailing<br>
> list and this is my first post :)<br>
><br>
> Recently, we've discovered that small-file performance on our system
is<br>
> pretty lack-luster. For comparison, here are some numbers:<br>
><br>
> 1) When transferring large files (~2 GB), we get outstanding<br>
> performance and can typically saturate the client's network connection.<br>
> We generally see about 490 MB/s over a 10Gb line, which should be
about<br>
> right, given that we lose half of our bandwidth to replication.<br>
><br>
> 2) When transferring a large number of small files, we get a
very poor<br>
> transfer rate, generally on the order of 2 MB/s, writing from a client<br>
> node *inside* the GPFS cluster.<br>
><br>
> I'm wondering if anyone else has experience with similar performance<br>
> issues and what ended up being the cause/solution. Also, I would
be<br>
> interested in hearing any general rules-of-thumb that the group has<br>
> found helpful in balancing performance between large-file and small-file<br>
> I/O.<br>
><br>
> We have gathered some diagnostic information while performing various<br>
> small-file I/O operations, as well as a variety of metadata operations<br>
> in quick succession. I'd be happy to share results of the diagnostics,<br>
> if it would help provide context.<br>
><br>
> Thank you so much for all of your help!<br>
><br>
> Stewart Howard<br>
> Indiana University<br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> gpfsug-discuss mailing list<br>
> gpfsug-discuss at gpfsug.org<br>
> </font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>
><br>
<br>
-- <br>
chekh@stanford.edu 347-401-4860<br>
_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at gpfsug.org<br>
</font></tt><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><font size=2>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</font></tt></a><tt><font size=2><br>
<br>
</font></tt>
<br>