[gpfsug-discuss] GPFS FPO

Aaron Knister aaron.s.knister at nasa.gov
Wed Aug 24 17:43:56 BST 2016


To tell you the truth, I don't. It's on my radar but I haven't done it 
yet. I *have* run hadoop on GPFS w/o magpie though and on only a couple 
of nodes was able to pound 1GB/s out to GPFS w/ the terasort benchmark. 
I know our GPFS FS can go much faster than that but java was cpu-bound 
as it often seems to be.

-Aaron

On 8/23/16 7:56 AM, Brian Marshall wrote:
> Aaron,
>
> Do you have experience running this on native GPFS?  The docs say Lustre
> and any NFS filesystem.
>
> Thanks,
> Brian
>
>
> On Aug 22, 2016 10:37 PM, "Aaron Knister" <aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>> wrote:
>
>     Yes, indeed. Note that these are my personal opinions.
>
>     It seems to work quite well and it's not terribly hard to set up or
>     get running. That said, if you've got a traditional HPC cluster with
>     reasonably good bandwidth (and especially if your data is already on
>     the HPC cluster) I wouldn't bother with FPO and just use something
>     like magpie (https://github.com/LLNL/magpie
>     <https://github.com/LLNL/magpie>) to run your hadoopy workload on
>     GPFS on your traditional HPC cluster. I believe FPO (and by
>     extension data locality) is important when the available bandwidth
>     between your clients and servers/disks (in a traditional GPFS
>     environment) is less than the bandwidth available within a node
>     (e.g. between your local disks and the host CPU).
>
>     -Aaron
>
>     On 8/22/16 10:23 PM, Brian Marshall wrote:
>
>         Does anyone have any experiences to share (good or bad) about
>         setting up
>         and utilizing FPO for hadoop compute on top of GPFS?
>
>
>         _______________________________________________
>         gpfsug-discuss mailing list
>         gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>         <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776 <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776



More information about the gpfsug-discuss mailing list