[gpfsug-discuss] GPFS FPO
Aaron Knister
aaron.s.knister at nasa.gov
Wed Aug 24 17:43:56 BST 2016
To tell you the truth, I don't. It's on my radar but I haven't done it
yet. I *have* run hadoop on GPFS w/o magpie though and on only a couple
of nodes was able to pound 1GB/s out to GPFS w/ the terasort benchmark.
I know our GPFS FS can go much faster than that but java was cpu-bound
as it often seems to be.
-Aaron
On 8/23/16 7:56 AM, Brian Marshall wrote:
> Aaron,
>
> Do you have experience running this on native GPFS? The docs say Lustre
> and any NFS filesystem.
>
> Thanks,
> Brian
>
>
> On Aug 22, 2016 10:37 PM, "Aaron Knister" <aaron.s.knister at nasa.gov
> <mailto:aaron.s.knister at nasa.gov>> wrote:
>
> Yes, indeed. Note that these are my personal opinions.
>
> It seems to work quite well and it's not terribly hard to set up or
> get running. That said, if you've got a traditional HPC cluster with
> reasonably good bandwidth (and especially if your data is already on
> the HPC cluster) I wouldn't bother with FPO and just use something
> like magpie (https://github.com/LLNL/magpie
> <https://github.com/LLNL/magpie>) to run your hadoopy workload on
> GPFS on your traditional HPC cluster. I believe FPO (and by
> extension data locality) is important when the available bandwidth
> between your clients and servers/disks (in a traditional GPFS
> environment) is less than the bandwidth available within a node
> (e.g. between your local disks and the host CPU).
>
> -Aaron
>
> On 8/22/16 10:23 PM, Brian Marshall wrote:
>
> Does anyone have any experiences to share (good or bad) about
> setting up
> and utilizing FPO for hadoop compute on top of GPFS?
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
> --
> Aaron Knister
> NASA Center for Climate Simulation (Code 606.2)
> Goddard Space Flight Center
> (301) 286-2776 <tel:%28301%29%20286-2776>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776
More information about the gpfsug-discuss
mailing list