[gpfsug-discuss] Maxblocksize tuning alternatives/max number of buffers

Fri Feb 28 21:53:25 GMT 2020

Hello Anderson,

This application requires minimum throughput of about 10-13MB/s initially and almost no IOPS during first phase where it opens all the files and reads the headers and about 30MB/s throughput during the second phase.
The issue that I face is during the second phase where it tries to randomly read about 4K of block size from random files from 20000 to about 100000.
In this phase - I see a big difference in maxblocksize parameter changing the performance of the reads, with almost no throughput and may be around 2-4K IOPS.

This issue is a follow up to the previous issue that I had mentioned about an year ago - where I see differences in performance - “though there is practically no IO to the storage”
I mean - I see a difference  in performance between different FS block-sizes even if all data is cached in pagepool.
Sven had replied to that thread mentioning that it could be because of buffer locking issue.

The info requested is as below: 

4 Storage clusters:

Storage cluster for compute:
5.0.3-2 GPFS version
FS version: 19.01 (5.0.1.0)
Subblock size: 16384
Blocksize : 16M 

Flash Storage Cluster for compute:
5.0.4-2 GPFS version
FS version: 18.00 (5.0.0.0)
Subblock size: 8192
Blocksize: 512K

Storage cluster for admin tools:
5.0.4-2 GPFS version
FS version: 16.00 (4.2.2.0)
Subblock size: 131072
Blocksize: 4M

Storage cluster for archival:
5.0.3-2 GPFS version
FS version: 16.00 (4.2.2.0)
Subblock size: 32K
Blocksize: 1M 

The only two clusters that users do/will do compute on is the 16M filesystem and the 512K Filesystem.

When you ask what is the throughput/IOPS and block size - it varies a lot and has not been recorded.
The 16M FS is capable of doing about 27GB/s seq read for about 1.8 PB of storage.
The 512K FS is capable of doing about 10-12GB/s seq read for about 100T of storage.

Now as I mentioned previously - the issue that I am seeing has been related to different FS block sizes on the same storage.
For example: 
On the Flash Storage cluster: 
Block size of 512K with maxblocksize of 16M gives worse performance than Block size of 512K with maxblocksize of 512K.
It is the maxblocksize that is affecting the performance, on the same storage with same block size and everything else being the same.
I am thinking the above is because of the number of buffers involved, but would like to learn if it happens to be anything else.
I have debugged the same with IBM GPFS techs and it has been found that there is no issue with the storage itself or any of the other GPFS tuning parameters.

Now since we do know that maxblocksize is making a big difference.
I would like to keep it as low as possible but still be able to mount other remote GPFS filesystems with higher block sizes.
Or since it is required to keep the maxblocksize the same across all storage - I would like to know if there is any other parameters that could do the same change as maxblocksize.

Thank you,
Lohit   

> On Feb 28, 2020, at 12:58 PM, Anderson Ferreira Nobre <anobre at br.ibm.com> wrote:
> 
> Hi Lohit,
>  
> First, a few questions to understand better your problem:
> - What is the minimum release level of both clusters?
> - What is the version of filesystem layout for 16MB, 1MB and 512KB?
> - What is the subblocksize of each filesystem?
> - How many IOPS, block size and throughput are you doing on each filesystem?
>  
> Abraços / Regards / Saludos,
>  
> Anderson Nobre
> Power and Storage Consultant
> IBM Systems Hardware Client Technical Team – IBM Systems Lab Services
> 
> 
>  
> Phone: 55-19-2132-4317
> E-mail: anobre at br.ibm.com <mailto:anobre at br.ibm.com>	
>  
>  
> ----- Original message -----
> From: "Valleru, Lohit/Information Systems" <valleru at cbio.mskcc.org>
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
> To: gpfsug-discuss at spectrumscale.org
> Cc:
> Subject: [EXTERNAL] [gpfsug-discuss] Maxblocksize tuning alternatives/max number of buffers
> Date: Fri, Feb 28, 2020 12:30
>  
> Hello Everyone,
> 
> I am looking for alternative tuning parameters that could do the same job as tuning the maxblocksize parameter.
> 
> One of our users run a deep learning application on GPUs, that does the following IO pattern:
> 
> It needs to read random small sections about 4K in size from about 20,000 to 100,000 files of each 100M to 200M size.
> 
> When performance tuning for the above application on a 16M filesystem and comparing it to various other file system block sizes - I realized that the performance degradation that I see might be related to the number of buffers.
> 
> I observed that the performance varies widely depending on what maxblocksize parameter I use.
> For example, using a 16M maxblocksize for a 512K or a 1M block size filesystem differs widely from using a 512K or 1M maxblocksize for a  512K or a 1M block size filesystem.
> 
> The reason I believe might be related to the number of buffers that I could keep on the client side, but I am not sure if that is the all that the maxblocksize is affecting.
> 
> We have different file system block sizes in our environment ranging from 512K, 1M and 16M.
> 
> We also use storage clusters and compute clusters design.
> 
> Now in order to mount the 16M filesystem along with the other filesystems on compute clusters - we had to keep the maxblocksize to be 16M - no matter what the file system block size.
> 
> I see that I get maximum performance for this application from a 512K block size filesystem and a 512K maxblocksize.
> However, I will not be able to mount this filesystem along with the other filesystems because I will need to change the maxblocksize to 16M in order to mount the other filesystems of 16M block size.
> 
> I am thinking if there is anything else that can do the same job as maxblocksize parameter.
> 
> I was thinking about the parameters like maxBufferDescs for a 16M maxblocksize, but I believe it would need a lot more pagepool to keep the same number of buffers as would be needed for a 512k maxblocksize.
> 
> May I know if there is any other parameter that could help me the same as maxblocksize, and the side effects of the same?
> 
> Thank you,
> Lohit
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss <http://gpfsug.org/mailman/listinfo/gpfsug-discuss> 
>  
>  
> 
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20200228/5a5e934c/attachment-0002.htm>