[gpfsug-discuss] GPFS and Flash/SSD Storage tiered storage

Jonathan Buzzard jonathan.buzzard at strath.ac.uk
Sat Feb 24 12:01:08 GMT 2018


On 23/02/18 01:27, valleru at cbio.mskcc.org wrote:
> Thanks, I will try the file heat feature but i am really not sure, if it 
> would work - since the code can access cold files too, and not 
> necessarily files recently accessed/hot files.
> 
> With respect to LROC. Let me explain as below:
> 
> The use case is that -
> The code initially reads headers (small region of data) from thousands 
> of files as the first step. For example about 30,000 of them with each 
> about 300MB to 500MB in size.
> After the first step, with the help of those headers - it mmaps/seeks 
> across various regions of a set of files in parallel.
> Since its all small IOs and it was really slow at reading from GPFS over 
> the network directly from disks - Our idea was to use AFM which i 
> believe fetches all file data into flash/ssds, once the initial few 
> blocks of the files are read.
> But again - AFM seems to not solve the problem, so i want to know if 
> LROC behaves in the same way as AFM, where all of the file data is 
> prefetched in full block size utilizing all the worker threads  - if few 
> blocks of the file is read initially.
> 

Imagine a single GPFS file system, metadata in SSD, a fast data pool and 
a slow data pool (fast and slow being really good names to avoid the 8 
character nonsense).

Now if your fast data pool is appropriately sized then your slow data 
pool will normally be doing diddly squat. We are talking under 10 I/O's 
per second. Frankly under 5 I/O's per second is more like it from my 
experience.

If your slow pool is 8-10PB in size, then it has thousands of spindles 
in it, and should be able to absorb the start of the job without 
breaking sweat. For numbers a 7.2K RPM disk can do around 120 random 
I/O's per second, so using RAID6 and 8TB disks that's 130 LUN's so 
around 15,000 random I/O's per second spare overhead, more if it's not 
random. It should take all of around 1-2s to read in those headers. 
Therefore unless these jobs only run for a few seconds or you have 
dozens of them starting every minute it should not be an issue.

Finally if GPFS is taking ages to read the files over the network, then 
it sounds like your network needs an upgrade or GPFS needs tuning which 
may or may not require a larger fast storage pool.


JAB.

-- 
Jonathan A. Buzzard                         Tel: +44141-5483420
HPC System Administrator, ARCHIE-WeSt.
University of Strathclyde, John Anderson Building, Glasgow. G4 0NG



More information about the gpfsug-discuss mailing list