[gpfsug-discuss] Confusing I/O Behavior

[Jonathan Buzzard]

On Tue, 2018-04-10 at 23:43 +0200, Uwe Falke wrote:
> Hi Aaron, 
> to how many different files do these tiny I/O requests go?
> 
> Mind that the write aggregates the I/O over a limited time (5 secs or
> so) and ***per file***. 
> It is for that matter a large difference to write small chunks all to
> one 
> file or to a large number of individual files .
> to fill a  1 MiB buffer you need about 13100 chunks of  80Bytes
> ***per 
> file*** within those 5 secs. 
> 

Something else to bear in mind is that you might be using a library
that converts everything into putchar's. I have seen this in the past
with Office on a Mac platform and made performance saving a file over
SMB/NFS appalling. I mean really really bad, a "save as" which didn't
do that would take a second or two, a save would take like 15 minutes.
To the local disk it was just fine.

The GPFS angle is this was all on a self rolled clustered Samba GPFS
setup back in the day. Took a long time to track down, and performance
turned out to be just as appalling with a real Windows file server.

JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG