<html><body><p><font size="2">Hi Oluwasijibomi,<br></font><font size="2"><br></font><font size="2">If you set up a Storage Insights Standard account <br></font><font size="2">You can monitor the performance of your 5030, and pull the performance metrics of the block storage array when you see poor performance in your scale cluster.<br></font><font size="2">This will give you some idea as to what is happening, <br></font><font size="2">but The 5030 is designed to be a backup / low IOPS Storage controller, the processing power and memory in the controllers is very limited.<br></font><font size="2"><br></font><font size="2">If you have significant workload happening on your file system in terms of user access (reads / writes) I am not at all surprised your seeing performance bottleneck from the 5030.<br></font><font size="2"><br></font><font size="2">You could ask you local IBM Presales team to perform a StorM disk model of the expected performance using your current configuration to show you what you performance should look like.<br></font><font size="2"><br></font><font size="2">Regards<br></font><font size="2"><br></font><font size="2">Andrew Beattie<br></font><font size="2">Technical Sales - Storage for Data and AI<br></font><font size="2">IBM Australia and New Zealand <br></font><font size="2"><br></font><font size="2">> On 29 May 2021, at 06:04, Uwe Falke <UWEFALKE@de.ibm.com> wrote:<br></font><font size="2">> <br></font><font size="2">> Hi, odd prefetch strategy would affect read performance, but write latency <br></font><font size="2">> is claimed to be even worse ...<br></font><font size="2">> Have you simply checked what the actual IO performance of the v5k box <br></font><font size="2">> under that load is and how it compares to its nominal performance and that <br></font><font size="2">> of its disks?<br></font><font size="2">> how is the storage organised? how many LUNs/NSDs, what RAID code (V5k <br></font><font size="2">> cannot do declustered RAID, can it?), any thin provisioning or other <br></font><font size="2">> gimmicks in the game?<br></font><font size="2">> what IO sizes ?<br></font><font size="2">> tons of things to look at. <br></font><font size="2">> <br></font><font size="2">> Mit freundlichen Grüßen / Kind regards<br></font><font size="2">> <br></font><font size="2">> Dr. Uwe Falke<br></font><font size="2">> IT Specialist<br></font><font size="2">> Hybrid Cloud Infrastructure / Technology Consulting & Implementation <br></font><font size="2">> Services<br></font><font size="2">> +49 175 575 2877 Mobile<br></font><font size="2">> Rochlitzer Str. 19, 09111 Chemnitz, Germany<br></font><font size="2">> uwefalke@de.ibm.com<br></font><font size="2">> <br></font><font size="2">> IBM Services<br></font><font size="2">> <br></font><font size="2">> IBM Data Privacy Statement<br></font><font size="2">> <br></font><font size="2">> IBM Deutschland Business & Technology Services GmbH<br></font><font size="2">> Geschäftsführung: Sven Schooss, Stefan Hierl<br></font><font size="2">> Sitz der Gesellschaft: Ehningen<br></font><font size="2">> Registergericht: Amtsgericht Stuttgart, HRB 17122<br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> From: Jan-Frode Myklebust <janfrode@tanso.net><br></font><font size="2">> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org><br></font><font size="2">> Date: 28/05/2021 19:50<br></font><font size="2">> Subject: [EXTERNAL] Re: [gpfsug-discuss] Long IO waiters and IBM <br></font><font size="2">> Storwize V5030<br></font><font size="2">> Sent by: gpfsug-discuss-bounces@spectrumscale.org<br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> One thing to check: Storwize/SVC code will *always* guess wrong on <br></font><font size="2">> prefetching for GPFS. You can see this with having a lot higher read data <br></font><font size="2">> throughput on mdisk vs. on on vdisks in the webui. To fix it, disable <br></font><font size="2">> cache_prefetch with "chsystem -cache_prefetch off".<br></font><font size="2">> <br></font><font size="2">> This being a global setting, you probably only should set it if the system <br></font><font size="2">> is only used for GPFS. <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> -jf<br></font><font size="2">> <br></font><font size="2">> On Fri, May 28, 2021 at 5:58 PM Saula, Oluwasijibomi <<br></font><font size="2">> oluwasijibomi.saula@ndsu.edu> wrote:<br></font><font size="2">> Hi Folks,<br></font><font size="2">> <br></font><font size="2">> So, we are experiencing some very long IO waiters in our GPFS cluster:<br></font><font size="2">> <br></font><font size="2">> # mmdiag --waiters <br></font><font size="2">> <br></font><font size="2">> === mmdiag: waiters ===<br></font><font size="2">> Waiting 17.3823 sec since 10:41:01, monitored, thread 21761 NSDThread: for <br></font><font size="2">> I/O completion<br></font><font size="2">> Waiting 16.6140 sec since 10:41:02, monitored, thread 21730 NSDThread: for <br></font><font size="2">> I/O completion<br></font><font size="2">> Waiting 15.3004 sec since 10:41:03, monitored, thread 21763 NSDThread: for <br></font><font size="2">> I/O completion<br></font><font size="2">> Waiting 15.2013 sec since 10:41:03, monitored, thread 22175 <br></font><font size="2">> <br></font><font size="2">> However, GPFS support is pointing to our IBM Storwize V5030 disk system as <br></font><font size="2">> the source of latency. Unfortunately, we don't have paid support for the <br></font><font size="2">> system so we are polling for anyone who might be able to assist.<br></font><font size="2">> <br></font><font size="2">> Does anyone by chance have any experience with IBM Storwize V5030 or <br></font><font size="2">> possess a problem determination guide for the V5030?<br></font><font size="2">> <br></font><font size="2">> We've briefly reviewed the V5030 management portal, but we still haven't <br></font><font size="2">> identified a cause for the increased latencies (i.e. read ~129ms, write <br></font><font size="2">> ~198ms). <br></font><font size="2">> <br></font><font size="2">> Granted, we have some heavy client workloads, yet we seem to experience <br></font><font size="2">> this drastic drop in performance every couple of months, probably <br></font><font size="2">> exacerbated by heavy IO demands.<br></font><font size="2">> <br></font><font size="2">> Any assistance would be much appreciated.<br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> Thanks,<br></font><font size="2">> <br></font><font size="2">> (Siji) Saula<br></font><font size="2">> HPC Systems Administrator / Information Technology<br></font><font size="2">> <br></font><font size="2">> Research 2 Building 220B / Fargo ND 58108-6050<br></font><font size="2">> p: 701.231.7749 / www.ndsu.edu<br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> _______________________________________________<br></font><font size="2">> gpfsug-discuss mailing list<br></font><font size="2">> gpfsug-discuss at spectrumscale.org<br></font><font size="2">> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br></font><font size="2">> _______________________________________________<br></font><font size="2">> gpfsug-discuss mailing list<br></font><font size="2">> gpfsug-discuss at spectrumscale.org<br></font><font size="2">> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> <br></font><font size="2">> _______________________________________________<br></font><font size="2">> gpfsug-discuss mailing list<br></font><font size="2">> gpfsug-discuss at spectrumscale.org<br></font><font size="2">> <a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a> <br></font><font size="2">> <br></font><BR>
</body></html>