From pinto at scinet.utoronto.ca Fri Nov 1 23:40:31 2019 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 1 Nov 2019 19:40:31 -0400 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= Message-ID: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> How can I force secondary processes to use the folder instructed by the -g option? I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 -P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From spectrumscale at kiranghag.com Sat Nov 2 03:35:32 2019 From: spectrumscale at kiranghag.com (KG) Date: Sat, 2 Nov 2019 09:05:32 +0530 Subject: [gpfsug-discuss] Windows + LDAP + Linux clients interop Message-ID: Hi Folks I need to integrate Windows 10 clients into Spectrum Scale cluster that will have Linux clients. - The Linux clients get authenticated via LDAP - The Windows 10 clients authenticate via AD - There would not be any folder with simultaneous Win/Lin access however the ACLS are required to protect file/directory access amongst users I would like to know how file permissions and overall interop would happen. Do I need to change Win10 client to authenticate via LDAP? OR can they use LDAP while connecting and using GPFS mounts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Nov 4 01:24:35 2019 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sun, 3 Nov 2019 20:24:35 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= In-Reply-To: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> References: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> Message-ID: Please show us the 2 or 3 mmbackup commands that you would like to run concurrently. Peeking into the script, I find: if [[ $scope == "inode-space" ]] then deviceSuffix="${deviceName}.${filesetName}" else deviceSuffix="${deviceName}" I believe mmbackup is designed to allow concurrent backup of different independent filesets within the same filesystem, Or different filesystems... And a single mmbackup instance can drive several TSM servers, which can be named with an option or in the dsm.sys file: # --tsm-servers TSMserver[,TSMserver...] # List of TSM servers to use instead of the servers in the dsm.sys file. From: Jaime Pinto To: gpfsug main discussion list Date: 11/01/2019 07:40 PM Subject: [EXTERNAL] [gpfsug-discuss] mmbackup ?g GlobalWorkDirectory not being followed Sent by: gpfsug-discuss-bounces at spectrumscale.org How can I force secondary processes to use the folder instructed by the -g option? I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 -P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8D_-g0uvUjmBxM3FoCM3Jru71qum_AlcQYOe2gaC9Iw&s=rKNspohw_8ulLhO5Epvqly_vRyiBLxylBWPNKkea2eU&e= ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8D_-g0uvUjmBxM3FoCM3Jru71qum_AlcQYOe2gaC9Iw&s=Yahjw-3p5PGqhgawsVqB2LuwZ151Fov398camDm4XwY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From pinto at scinet.utoronto.ca Mon Nov 4 01:56:35 2019 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sun, 3 Nov 2019 20:56:35 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= In-Reply-To: References: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> Message-ID: <5bf94749-4add-c4f4-63df-21551c5111e1@scinet.utoronto.ca> On 11/3/2019 20:24:35, Marc A Kaplan wrote: > Please show us the 2 or 3 mmbackup commands that you would like to run concurrently. Hey Marc, They would be pretty similar, with the only different being the target TSM server, determined by sourcing a different dsmenv1(2 or 3) prior to the start of each instance, each with its own dsm.sys (3 wrappers). (source dsmenv1; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg1 --scope inodespace -v -a 8 -L 2) (source dsmenv3; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg2 --scope inodespace -v -a 8 -L 2) (source dsmenv3; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg3 --scope inodespace -v -a 8 -L 2) I was playing with the -L (to control the policy), but you bring up a very good point I had not experimented with, such as a single traverse for multiple target servers. It may be just what I need. I'll try this next. Thank you very much, Jaime > > Peeking into the script, I find: > > if [[ $scope == "inode-space" ]] > then > deviceSuffix="${deviceName}.${filesetName}" > else > deviceSuffix="${deviceName}" > > > I believe mmbackup is designed to allow concurrent backup of different independent filesets within the same filesystem, Or different filesystems... > > And a single mmbackup instance can drive several TSM servers, which can be named with an option or in the dsm.sys file: > > # --tsm-servers TSMserver[,TSMserver...] > # List of TSM servers to use instead of the servers in the dsm.sys file. > > > > Inactive hide details for Jaime Pinto ---11/01/2019 07:40:47 PM---How can I force secondary processes to use the folder instrucJaime Pinto ---11/01/2019 07:40:47 PM---How can I force secondary processes to use the folder instructed by the -g option? I started a mmbac > > From: Jaime Pinto > To: gpfsug main discussion list > Date: 11/01/2019 07:40 PM > Subject: [EXTERNAL] [gpfsug-discuss] mmbackup ?g GlobalWorkDirectory not being followed > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > > > How can I force secondary processes to use the folder instructed by the -g option? > > I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) > > However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. > > See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): > > DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 ?-P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace > > > Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. > > Thanks > Jaime > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From heinrich.billich at id.ethz.ch Mon Nov 4 17:13:50 2019 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Mon, 4 Nov 2019 17:13:50 +0000 Subject: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Message-ID: Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Nov 4 19:21:26 2019 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 4 Nov 2019 21:21:26 +0200 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=Mg9Yxn6axX4-iDSDZNIhF58cDheSv41MfR8uVXS_q58&s=bwXBF_0oFwkRREwv9IUvhXGgbQtjhEnJR5Xma7_XFIU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From sadaniel at us.ibm.com Mon Nov 4 19:43:18 2019 From: sadaniel at us.ibm.com (Steven Daniels) Date: Mon, 4 Nov 2019 12:43:18 -0700 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=LfJIsA_kqUNu0c81SWoB1avoQI9WhsKRAILIZcOW8ts&s=0CsyJjJK2f21MXg1WtTCrGNLrg31sqbejXCMQzLyNQo&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From YARD at il.ibm.com Tue Nov 5 11:42:34 2019 From: YARD at il.ibm.com (Yaron Daniel) Date: Tue, 5 Nov 2019 13:42:34 +0200 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: Hi Steven Can you please try to run mmlsnsd - and let me know if you can change the nsd name before/after configure them with mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Steven Daniels" To: gpfsug main discussion list Date: 04/11/2019 21:44 Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=EzTElY1rKKm1TuasJlFc6kJOA5qcwp0FPH71ofd8CsA&s=7Tbbpw_RbqYWzzoaTVvo7V_11FP9ytTQRHw_TWgC24Q&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From scale at us.ibm.com Tue Nov 5 13:51:45 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 5 Nov 2019 08:51:45 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: As far as I know there is no way to change nsd names once they have been created, and yes mmvdisk automatically generates the nsd names. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/05/2019 06:42 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Steven Can you please try to run mmlsnsd - and let me know if you can change the nsd name before/after configure them with mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Steven Daniels" To: gpfsug main discussion list Date: 04/11/2019 21:44 Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=JZEHO0Ia1aoBD_GuPcVjZWgvVTYFsFL5YXqg-0nZCbw&s=iN3GzQg9lDBGVOeJR99XKrc7spI5ZQWGat6gNX5bN8s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Tue Nov 5 18:04:30 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 5 Nov 2019 18:04:30 +0000 Subject: [gpfsug-discuss] Reminder: Sign up for the SSUG meeting at SC19 if you plan to attend Message-ID: Reminder for all those going to SC19 and plan on attending the Spectrum Scale User Group meeting on Sunday November 17th - please pre-register! Agenda details and registration link is here: https://www.spectrumscaleug.org/event/spectrum-scale-user-group-meeting-sc19/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Nov 6 13:06:56 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Wed, 6 Nov 2019 13:06:56 +0000 Subject: [gpfsug-discuss] CIUK Call for Speakers Message-ID: Hi All, It?s just a month until CIUK and as in the past we?re running a Spectrum Scale user group as part of the conference. We?ll be running in the morning of 5th December and are looking for a couple of speakers who could do user talks for the meeting. Please let me know if you are interested in doing a site update/talk on something there! As usual, for this user group you must be a registered CIUK attendee. Thanks Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Fri Nov 8 16:25:24 2019 From: damir.krstic at gmail.com (Damir Krstic) Date: Fri, 8 Nov 2019 10:25:24 -0600 Subject: [gpfsug-discuss] gpfs client and turning off swap Message-ID: I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. Let me know when you get a chance. Thank you. Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_johnson at brown.edu Fri Nov 8 16:31:20 2019 From: david_johnson at brown.edu (david_johnson at brown.edu) Date: Fri, 8 Nov 2019 11:31:20 -0500 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: References: Message-ID: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> We have most of our clients network booted and diskless ? no swap possible. Gpfs still works until someone runs the node out of memory.... -- ddj Dave Johnson > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > ? > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > Let me know when you get a chance. > > Thank you. > Damir > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From damir.krstic at gmail.com Fri Nov 8 16:37:03 2019 From: damir.krstic at gmail.com (Damir Krstic) Date: Fri, 8 Nov 2019 10:37:03 -0600 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: Thanks - that's what I thought. We also (many refreshes ago) ran diskless images with RedHat 6 and GPFS 3.4 and ran into no issues with having swap off. I figured to ask in case something has changed with the Spectrum Scale. Damir On Fri, Nov 8, 2019 at 10:31 AM wrote: > We have most of our clients network booted and diskless ? no swap > possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic > wrote: > > > > ? > > I was wondering if it's safe to turn off swap on gpfs client machines? > we have a case where checkpointing is swapping and we would like to prevent > it from doing so by disabling swap. However, the gpfs manual admin. manual > states to have swap enabled and sufficiently large, but it does not > distinguish between clients and IO servers. > > > > Let me know when you get a chance. > > > > Thank you. > > Damir > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Nov 8 16:44:47 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 8 Nov 2019 16:44:47 +0000 Subject: [gpfsug-discuss] SSUG meeting at SC19 - Important Updates! Message-ID: <6BEA6CAB-A84E-4C1E-8A56-F600F04C4082@nuance.com> For those of you going to (or planning on going) to the User Group meeting at SC19, here is some important updates: You should pre-register here: https://www.spectrumscaleug.org/event/spectrum-scale-user-group-meeting-sc19/ Box Lunches: Thanks to the generous supports of Mark III Systems and Lenovo, we will have box lunches available. These should arrive about 11:30AM on Sunday 11/17. There is a limited number (100) and will include a variety of sandwiches, and some vegetarian selections. Drinks are NOT included. First come, first serve. These are free of charge, and are available to any of those attending either the morning or afternoon session(s). Wi-Fi: Despite the best efforts of the Principals, Sponsors, and IBM, we haven?t been able to make this work. There WILL be a cable drop to the podium for presentations, but no public Wi-Fi access - Hotel Guest Wi-Fi should be available. Looking forward to seeing everyone at SC19! Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Fri Nov 8 16:49:21 2019 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Fri, 8 Nov 2019 16:49:21 +0000 Subject: [gpfsug-discuss] GPFS diskless/stateless clients (was: Re: gpfs client and turning off swap) In-Reply-To: References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: <229B3495-6D3D-40DF-8927-0960F751749C@rutgers.edu> We run stateless machines today with CentOS 7.x and have run GPFS 4.1.0.8, 4.2.3.x, and 5.0.x. We don?t use swap either in most cases, and have never needed to do anything special for that. We?ve generally needed to rig ExecStartPre stuff in systemd in order to make sure that the machines could restore their cluster membership. If anyone else can share how they do that, it might be interesting information. In our case, this works (our provisioning system writes these variables as the actual host names) ? we used to put this in an override to gpfs.service; I believe mmautoload.service is relatively new (either in 4.2.3.x to 5.0.x or 4.1.x to 4.2.x). [root at test mmautoload.service.d]# pwd /etc/systemd/system/mmautoload.service.d [root at test mmautoload.service.d]# cat override.conf [Unit] Before=slurmd.service After=sys-subsystem-net-devices-ib0.device [Service] ExecStartPre=/usr/lpp/mmfs/bin/mmsdrrestore -p $CLUSTERMGR -R /usr/bin/scp ExecStartPre=/usr/lpp/mmfs/bin/mmauth genkey propagate -N $NODENAME -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' > On Nov 8, 2019, at 11:37 AM, Damir Krstic wrote: > > Thanks - that's what I thought. We also (many refreshes ago) ran diskless images with RedHat 6 and GPFS 3.4 and ran into no issues with having swap off. I figured to ask in case something has changed with the Spectrum Scale. > Damir > > On Fri, Nov 8, 2019 at 10:31 AM wrote: > We have most of our clients network booted and diskless ? no swap possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > > > ? > > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > > > Let me know when you get a chance. > > > > Thank you. > > Damir > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From skylar2 at uw.edu Fri Nov 8 16:52:33 2019 From: skylar2 at uw.edu (Skylar Thompson) Date: Fri, 8 Nov 2019 16:52:33 +0000 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> We don't run diskless systems, but use cgroups where possible to limit the damage users can do to a node. We derate the node's total usable memory by 2GB (OS) + GPFS pagepool to avoid paging whenever possible. On Fri, Nov 08, 2019 at 11:31:20AM -0500, david_johnson at brown.edu wrote: > We have most of our clients network booted and diskless ??? no swap possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > > > ??? > > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > > > Let me know when you get a chance. -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From valdis.kletnieks at vt.edu Fri Nov 8 21:15:12 2019 From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks) Date: Fri, 08 Nov 2019 16:15:12 -0500 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: References: Message-ID: <30980.1573247712@turing-police> On Fri, 08 Nov 2019 10:25:24 -0600, Damir Krstic said: > I was wondering if it's safe to turn off swap on gpfs client machines? we > have a case where checkpointing is swapping and we would like to prevent it > from doing so by disabling swap. However, the gpfs manual admin. GPFS will work just fine without a swap space if the system has sufficient RAM. However, if checkpointing is swapping, you don't have enough RAM (at least with your current configuration), and turning off swap will result in processes being killed due to lack of memory. You *might* get it to work by tuning system config variables to reduce RAM consumption. But that's better tested while you still have swap defined, and not removing swap until you have a system not needing it. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Sat Nov 9 09:27:10 2019 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 9 Nov 2019 09:27:10 +0000 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> Message-ID: On 08/11/2019 16:52, Skylar Thompson wrote: > We don't run diskless systems, but use cgroups where possible to limit the > damage users can do to a node. We derate the node's total usable memory by > 2GB (OS) + GPFS pagepool to avoid paging whenever possible. > I would add that as the RAM in the node rises swap becomes pointless. We find that it's a bit pointless at 192GB of RAM in a node, for our large memory nodes which have 3TB of RAM, swap is utterly pointless. In fact there is more RAM than there is disk space. In either case the old rules about setting swap to be twice RAM or even swap equals RAM are either hard to achieve to impossible. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From julian.jakobs at cec.mpg.de Mon Nov 11 11:25:29 2019 From: julian.jakobs at cec.mpg.de (Jakobs, Julian) Date: Mon, 11 Nov 2019 11:25:29 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 Message-ID: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Hello, has anyone already tried Spectrum Scale on RHEL 8.1? I can see from the GPFS FAQ that "RHEL 8" (no minor version indicated) is supported as of the 5.0.4 release. However latest kernel level fully tested is 4.18.0-80, indicating that only RHEL 8.0 was tested. I tested an installation with 8.1 (kernel version 4.18.0-147) and mmbuildgpl failed due to errors while compiling the gpl (incompatible pointer type). Is this expected behaviour or is there maybe something else wrong with the installation? If this needs a new GPFS release, is there an estimated time? I would much prefer to install it with RHEL 8.1 due to 8.0 not being a EUS release. Best, Julian Jakobs -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6220 bytes Desc: not available URL: From stockf at us.ibm.com Mon Nov 11 12:47:28 2019 From: stockf at us.ibm.com (Frederick Stock) Date: Mon, 11 Nov 2019 12:47:28 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> References: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Nov 11 13:25:27 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 11 Nov 2019 13:25:27 +0000 Subject: [gpfsug-discuss] Force removal of problematic inode? Message-ID: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> Any ideas on how I could get out of this, short of a full mmfsck? Directory paths shortened to obfuscate data. [root at nrg5-gpfs01 remove]# tsfindinode -i 11198927 /gpfs/lm gpfs_iopen(/gpfs/lm/.../build_respell_map)(184769551:0xD4F8): Device or resource busy ^C ls -li /gpfs/lm/?/ ls: cannot access /gpfs/lm/.../build_respell_map: Device or resource busy total 0 ? d????????? ? ? ? ? ? build_respell_map [root at nrg5-gpfs01 remove]# rm -rf /gpfs/lm/.../* rm: cannot remove ?/gpfs/lm/.../build_respell_map?: Device or resource busy Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Mon Nov 11 13:28:19 2019 From: stockf at us.ibm.com (Frederick Stock) Date: Mon, 11 Nov 2019 13:28:19 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: References: , <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: From A.Wolf-Reber at de.ibm.com Mon Nov 11 13:30:09 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Mon, 11 Nov 2019 13:30:09 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: References: , <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746414.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746415.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746416.png Type: image/png Size: 1134 bytes Desc: not available URL: From alexander.zimmermann at id.ethz.ch Mon Nov 11 14:16:34 2019 From: alexander.zimmermann at id.ethz.ch (Zimmermann Alexander (ID SD)) Date: Mon, 11 Nov 2019 14:16:34 +0000 Subject: [gpfsug-discuss] orphaned deleted fileset Message-ID: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> Hi, we've an orphaned deleted fileset in one of our filesystems: [azi4ea at nas22ems01 ~]$ mmlsfileset /dev/fs2201 --deleted Filesets in file system 'fs2201': Name Status Path (chab_services_admin_s1) Deleted /fs2201/.snapshots/@GMT-2019.10.21-10.45.55/chab_services_admin_s1 (biol_bc_azzalin_1) Deleted -- [azi4ea at nas22ems01 ~]$ biol_bc_azzalin_1 is now in that state for roughly 1 week, initially it was in the state Deleting until I ran (another?) mmdelfileset command on it. The order to delete it in our automation tool was completed in July so it's hard find out what went wrong back then. Will it get cleaned up on its own and I'm only too impatient or do I need to intervene to get rid of it completely? Many thanks in advance for your aid! Kind regards, Alexander Zimmermann Alexander Zimmermann ETH Z?rich ID Systemdienste - Speicher Weinbergstrasse 11, WEC C16 8092 Z?rich Tel.: +41 44 632 72 39 E-Mail: alexander.zimmermann at id.ethz.ch www.id.ethz.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From will.schmied at stjude.org Mon Nov 11 22:02:19 2019 From: will.schmied at stjude.org (Schmied, Will) Date: Mon, 11 Nov 2019 22:02:19 +0000 Subject: [gpfsug-discuss] Job opportunity: HPC Storage Architect at St. Jude Message-ID: <1E710D36-37F5-4BCC-BE23-8A3A192B67BA@stjude.org> Hello everyone, St. Jude Children?s Research Hospital (Memphis, TN) has recently posted a job opening for a HPC Storage Architect, a senior level position working primarily to operate and maintain multiple Spectrum Scale clusters in support of research and other HPC workloads. You can view the job posting, and begin your application, here: http://myjob.io/nd6qd You can find all jobs, and information about working at St. Jude, here: https://www.stjude.org/jobs/hospital.html Please feel free to contact me directly off list if you have any questions. I?ll be at SC this year, and also presenting at the user group meeting, and hope to see you there. Thanks, Will ________________________________ Email Disclaimer: www.stjude.org/emaildisclaimer Consultation Disclaimer: www.stjude.org/consultationdisclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Tue Nov 12 08:16:29 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 12 Nov 2019 16:16:29 +0800 Subject: [gpfsug-discuss] Force removal of problematic inode? In-Reply-To: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> References: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> Message-ID: You can try if restarting GPFS daemon would help. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 11/11/2019 09:28 PM Subject: [EXTERNAL] [gpfsug-discuss] Force removal of problematic inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org Any ideas on how I could get out of this, short of a full mmfsck? Directory paths shortened to obfuscate data. [root at nrg5-gpfs01 remove]# tsfindinode -i 11198927 /gpfs/lm gpfs_iopen(/gpfs/lm/.../build_respell_map)(184769551:0xD4F8): Device or resource busy ^C ls -li /gpfs/lm/?/ ls: cannot access /gpfs/lm/.../build_respell_map: Device or resource busy total 0 ? d????????? ? ? ? ? ? build_respell_map [root at nrg5-gpfs01 remove]# rm -rf /gpfs/lm/.../* rm: cannot remove ?/gpfs/lm/.../build_respell_map?: Device or resource busy Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=Pid3Qt9isGIuPoObkvy-RLwrhUmEM-mNU4YAfNWpDRo&s=TMRis9DxrIhg7D9xwf4IYi0tNH1RrWqmeSL1cDD1hi4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From scale at us.ibm.com Tue Nov 12 08:18:24 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 12 Nov 2019 16:18:24 +0800 Subject: [gpfsug-discuss] orphaned deleted fileset In-Reply-To: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> References: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> Message-ID: run mmfsck or try to delete another fileset to see if that could trigger cleanup. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Zimmermann Alexander (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Date: 11/11/2019 10:16 PM Subject: [EXTERNAL] [gpfsug-discuss] orphaned deleted fileset Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, we?ve an orphaned deleted fileset in one of our filesystems: [azi4ea at nas22ems01 ~]$ mmlsfileset /dev/fs2201 --deleted Filesets in file system 'fs2201': Name Status Path (chab_services_admin_s1) Deleted /fs2201/.snapshots/@GMT-2019.10.21-10.45.55/chab_services_admin_s1 (biol_bc_azzalin_1) Deleted -- [azi4ea at nas22ems01 ~]$ biol_bc_azzalin_1 is now in that state for roughly 1 week, initially it was in the state Deleting until I ran (another?) mmdelfileset command on it. The order to delete it in our automation tool was completed in July so it?s hard find out what went wrong back then. Will it get cleaned up on its own and I?m only too impatient or do I need to intervene to get rid of it completely? Many thanks in advance for your aid! Kind regards, Alexander Zimmermann Alexander Zimmermann ETH Z?rich ID Systemdienste - Speicher Weinbergstrasse 11, WEC C16 8092 Z?rich Tel.: +41 44 632 72 39 E-Mail: alexander.zimmermann at id.ethz.ch www.id.ethz.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=dy9kUmE1i9dh2VjQBu8Cqis2Us0lfMxVc_OzA2Jwhfk&s=eazf8OnQPmH23k5Z0VwYOGDLa5-LL88CcvdUqctGMIY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From S.J.Thompson at bham.ac.uk Tue Nov 12 14:49:07 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Tue, 12 Nov 2019 14:49:07 +0000 Subject: [gpfsug-discuss] SC19 SSUG Lightning talks Message-ID: <5C96F4C4-13B8-41A7-8834-9D32928AE738@bham.ac.uk> Hi All, It?s less than a week until the Spectrum Scale user group at SC19 in Denver! As part of the agenda, we have included a lightning talk slot and I?m looking for people to volunteer to do a zero slide talk on something related to Scale. Those at the UK event may remember we ran something similar on ?my favourite/worst feature of Spectrum Scale?. I?m happy to use the same topic, or for people to do a 3-5 minute lightning slot on Scale and why they use it ? or some other Scale topic entirely! If you are interested, I?ll be around on Sunday at the SSUG and you can let me know then/volunteer in the slot. If no-one volunteers, we?ll have a very quiet 30 mins in the user group session! Please have a think on something you might want to say on the day! Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From Rafael.Cezario at ibm.com Wed Nov 13 17:24:34 2019 From: Rafael.Cezario at ibm.com (Rafael Cezario) Date: Wed, 13 Nov 2019 14:24:34 -0300 Subject: [gpfsug-discuss] Online migration Message-ID: Hello, I have a Spectrum Scale cluster with 10 nodes on the version 4.2. The cluster have maxblocksize=DEFAULT. I will migrate all nodes to 5.0.3 (online) none downtime. But the maxblocksize is fixed on the mmlsconfig after the migration (the man have this information) maxblocksize 1M man mmchconfig Note: When you migrate a cluster from an earlier version to 5.0.0 or later, the value of maxblocksize stays the same. However, if maxblocksize was set to DEFAULT in the earlier version of the cluster, then migrating it to 5.0.0 or later sets it explicitly to 1 MiB, which was the default value in earlier versions. To change maxblocksize to the default value after migrating to 5.0.0 or later, set maxblocksize=DEFAULT (4 MiB). How Can I perform non-stop migration and after migration create a new filesystem with blocksize = 4MiB? My question is about finishing the migration and create a new filesystem with block size set to 4 MiB, its won't be possible until the maxblocksize changes. In a environment where all migration will be done without stopping the cluster, my question was if there was any alternative This behavior forces the entire cluster to stop. Regards, Rafael Cez?rio Dam?sio Cunha IT Specialist - PowerVM/AIX/Spectrum Scale Technology Support Services Global Technology Services Mobile: 55-6198122-4435 E-mail: rafael.cezario at ibm.com Scn Quadra 4, Bloco B, n.100 Brasilia, DF 70714-900 Brazil -------------- next part -------------- An HTML attachment was scrubbed... URL: From knop at us.ibm.com Wed Nov 13 19:05:00 2019 From: knop at us.ibm.com (Felipe Knop) Date: Wed, 13 Nov 2019 19:05:00 +0000 Subject: [gpfsug-discuss] Online migration In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From andi at christiansen.xxx Wed Nov 13 21:33:57 2019 From: andi at christiansen.xxx (Andi Christiansen) Date: Wed, 13 Nov 2019 22:33:57 +0100 (CET) Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. Message-ID: <384482306.75861.1573680837216@privateemail.com> An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Nov 13 22:05:08 2019 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 13 Nov 2019 22:05:08 +0000 Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. In-Reply-To: <384482306.75861.1573680837216@privateemail.com> References: <384482306.75861.1573680837216@privateemail.com> Message-ID: Hello, This is simply due to the fact that your ?test? is doing parallel appended writes to the same file. With GPFS, there is one node at a time that can write to roughly the same position (aka byte range) in a file. This is handled by something called a write_lock token that has to be revoked on the node that currently owns the token, then given to one of the other nodes attempting to perform the write. This overhead in lock token management is done for almost every write that is happening in this ?test?. This is not something that you would ever really want an application to do anyways. GPFS offers byte-range locking to allow for very fast, non-blocking I/O to the same file from multiple nodes in parallel. So if you change your ?test? so that each node seeks to a different part of the single file and that each node writes into the file in a way that there is no overlap in the byte range of your total writes from the node, then GPFS will give a byte-range write_lock token to each node. Since no node will attempt to write to the same part of the file as another node (which is what you?d want anyways in most cases) then there will be no write_lock token revoking and no blocking of your I/O to the single file. Hope that helps, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Andi Christiansen Sent: Wednesday, November 13, 2019 3:34 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. [EXTERNAL EMAIL] Hi all, I am in the process of replacing a beegfs cluster with a spectrum scale cluster and some of our initial tests have returned poor performance when writing from multiple client nodes to the same file. If we setup a client to write into a file it takes less than 8 seconds to complete the write on Flash and about the same on NLSAS storage. But if we get 3 more nodes to do the exact same write the cluster seems to slow down and completes all writes in about 1.5 minute. We are running 5.0.4-0 on 4 Lenovo SR630 servers with a V7000 control enclosure with flash drives and a 92F drawer with NLSAS drives behind the nodes attach through SAN. Is there something I am missing since the writes are so slow compared to beegfs? Beegfs completes the write from all clients within 9 second when run from 4 clients doing the same write to the same file GPFS takes 1.5 min to do the same. Tests run: time (for i in `seq 5000`;do echo "This is $(hostname) writing line number" $i >> "/gpfs_T0/test/benchmark1.txt";done) ########## this is run from 4 gpfs client nodes at the same time. Result for scale: real 1m43.995s user 0m0.821s sys 0m3.545s Result for beegfs: real 0m6.507s user 0m0.651s sys 0m2.534s if we run writes from each client node to separate files, performance is way better with GPFS than beegfs but not when the writes are done parallel. If anyone have an idea I would be glad to hear it ? Best Regards Andi Christiansen -------------- next part -------------- An HTML attachment was scrubbed... URL: From tshukla at NYGENOME.ORG Wed Nov 20 15:36:16 2019 From: tshukla at NYGENOME.ORG (Tushit Shukla) Date: Wed, 20 Nov 2019 15:36:16 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 81, Issue 43 In-Reply-To: References: Message-ID: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> Lohit, Did you get any progress on file system performance issue with large block size ? Apparently we are also going through the same issue after we migrated to 16M file system (from 4MB filesystem). Only thing which gives some relief is when we increase pagepool to 32G. We also tried playing with prefetchAggressivenessRead parameter by reducing it to 1 but not sure if it is doing anything. thanks ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 22, 2018 12:18:58 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 81, Issue 43 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: GPFS, Pagepool and Block size -> Perfomance reduces with larger block size (Sven Oehme) ---------------------------------------------------------------------- Message: 1 Date: Mon, 22 Oct 2018 09:18:43 -0700 From: Sven Oehme To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS, Pagepool and Block size -> Perfomance reduces with larger block size Message-ID: Content-Type: text/plain; charset="utf-8" oops, somehow that slipped my inbox, i only saw that reply right now. its really hard to see from a trace snipped if the lock is the issue as the lower level locks don't show up in default traces. without having access to source code and a detailed trace you won't make much progress here. sven On Thu, Sep 27, 2018 at 12:31 PM wrote: > Thank you Sven, > > Turning of prefetching did not improve the performance, but it did degrade > a bit. > > I have made the prefetching default and took trace dump, for tracectl with > trace=io. Let me know if you want me to paste/attach it here. > > May i know, how could i confirm if the below is true? > > 1. this could be serialization around buffer locks. as larger your >>> blocksize gets as larger is the amount of data one of this pagepool buffers >>> will maintain, if there is a lot of concurrency on smaller amount of data >>> more threads potentially compete for the same buffer lock to copy stuff in >>> and out of a particular buffer, hence things go slower compared to the same >>> amount of data spread across more buffers, each of smaller size. >>> >>> > Will the above trace help in understanding if it is a serialization issue? > > I had been discussing the same with GPFS support for past few months, and > it seems to be that most of the time is being spent at cxiUXfer. They could > not understand on why it is taking spending so much of time in cxiUXfer. I > was seeing the same from perf top, and pagefaults. > > Below is snippet from what the support had said : > > ???????????????????????????? > > I searched all of the gpfsRead from trace and sort them by spending-time. > Except 2 reads which need fetch data from nsd server, the slowest read is > in the thread 72170. It took 112470.362 us. > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum: 72165 6.860911319 > rdwr 141857.076 us + NSDIO > > trcrpt.2018-08-06_12.26.28.39794.lt15.trsum: 72170 1.483947593 > rdwr 112470.362 us + cxiUXfer > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum: 72165 6.949042593 > rdwr 88126.278 us + NSDIO > > trcrpt.2018-08-06_12.27.03.47706.lt15.trsum: 72156 2.919334474 > rdwr 81057.657 us + cxiUXfer > > trcrpt.2018-08-06_12.23.30.72745.lt15.trsum: 72154 1.167484466 > rdwr 76033.488 us + cxiUXfer > > trcrpt.2018-08-06_12.24.06.7508.lt15.trsum: 72187 0.685237501 > rdwr 70772.326 us + cxiUXfer > > trcrpt.2018-08-06_12.25.17.23989.lt15.trsum: 72193 4.757996530 > rdwr 70447.838 us + cxiUXfer > > > I check each of the slow IO as above, and find they all spend much time in > the function cxiUXfer. This function is used to copy data from kernel > buffer to user buffer. I am not sure why it took so much time. This should > be related to the pagefaults and pgfree you observed. Below is the trace > data for thread 72170. > > > 1.371477231 72170 TRACE_VNODE: gpfs_f_rdwr enter: fP > 0xFFFF882541649400 f_flags 0x8000 flags 0x8001 op 0 iovec > 0xFFFF881F2AFB3E70 count 1 offset 0x168F30D dentry 0xFFFF887C0CC298C0 > private 0xFFFF883F607175C0 iP 0xFFFF8823AA3CBFC0 name '410513.svs' > > .... > > 1.371483547 72170 TRACE_KSVFS: cachedReadFast exit: > uio_resid 16777216 code 1 err 11 > > .... > > 1.371498780 72170 TRACE_KSVFS: kSFSReadFast: oiP > 0xFFFFC90060B46740 offset 0x168F30D dataBufP FFFFC9003645A5A8 nDesc 64 buf > 200043C0000 valid words 64 dirty words 0 blkOff 0 > > 1.371499035 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate begin ul 0xFFFFC900333F1A40 holdCount 0 > ioType 0x2 inProg 0x15 > > 1.371500157 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate ul 0xFFFFC900333F1A40 holdCount 0 ioType 0x2 > inProg 0x16 err 0 > > 1.371500606 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 toIOBuf 0 offset 6877965 len > 9899251 > > 1.371500793 72170 TRACE_KSVFS: cxiUXfer: ndesc 0 skip > dataAddrP 0x200043C0000 currOffset 0 currLen 262144 bufOffset 6877965 > > .... > > 1.371505949 72170 TRACE_KSVFS: cxiUXfer: ndesc 25 skip > dataAddrP 0x2001AF80000 currOffset 6553600 currLen 262144 bufOffset 6877965 > > 1.371506236 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > currOffset 6815744 tmpLen 262144 dataAddrP 0x2001AFCF30D currLen 199923 > pageOffset 781 pageLen 3315 plP 0xFFFF887F7B90D600 > > 1.373649823 72170 TRACE_KSVFS: cxiUXfer: nDesc 27 > currOffset 7077888 tmpLen 262144 dataAddrP 0x20027400000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.375158799 72170 TRACE_KSVFS: cxiUXfer: nDesc 28 > currOffset 7340032 tmpLen 262144 dataAddrP 0x20027440000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.376661566 72170 TRACE_KSVFS: cxiUXfer: nDesc 29 > currOffset 7602176 tmpLen 262144 dataAddrP 0x2002C180000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.377892653 72170 TRACE_KSVFS: cxiUXfer: nDesc 30 > currOffset 7864320 tmpLen 262144 dataAddrP 0x2002C1C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > .... > > 1.471389843 72170 TRACE_KSVFS: cxiUXfer: nDesc 62 > currOffset 16252928 tmpLen 262144 dataAddrP 0x2001D2C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.471845629 72170 TRACE_KSVFS: cxiUXfer: nDesc 63 > currOffset 16515072 tmpLen 262144 dataAddrP 0x2003EC80000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.472417149 72170 TRACE_KSVFS: cxiDetachIOBuffer: > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 > > 1.472417775 72170 TRACE_LOCK: unlock_vfs: type Data, > key 0000000000000004:000000001B1F24BF:0000000000000001 lock_mode have ro > token xw lock_state old [ ro:27 ] new [ ro:26 ] holdCount now 27 > > 1.472418427 72170 TRACE_LOCK: hash tab lookup vfs: > found cP 0xFFFFC9005FC0CDE0 holdCount now 14 > > 1.472418592 72170 TRACE_LOCK: lock_vfs: type Data key > 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode want ro status > valid token xw/xw lock_state [ ro:12 ] flags 0x0 holdCount 14 > > 1.472419842 72170 TRACE_KSVFS: kSFSReadFast: oiP > 0xFFFFC90060B46740 offset 0x2000000 dataBufP FFFFC9003643C908 nDesc 64 buf > 38033480000 valid words 64 dirty words 0 blkOff 0 > > 1.472420029 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate begin ul 0xFFFFC9005FC0CF98 holdCount 0 > ioType 0x2 inProg 0xC > > 1.472420187 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate ul 0xFFFFC9005FC0CF98 holdCount 0 ioType 0x2 > inProg 0xD err 0 > > 1.472420652 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 toIOBuf 0 offset 0 len 6877965 > > 1.472420936 72170 TRACE_KSVFS: cxiUXfer: nDesc 0 > currOffset 0 tmpLen 262144 dataAddrP 0x38033480000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.472824790 72170 TRACE_KSVFS: cxiUXfer: nDesc 1 > currOffset 262144 tmpLen 262144 dataAddrP 0x380334C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.473243905 72170 TRACE_KSVFS: cxiUXfer: nDesc 2 > currOffset 524288 tmpLen 262144 dataAddrP 0x38024280000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > .... > > 1.482949347 72170 TRACE_KSVFS: cxiUXfer: nDesc 24 > currOffset 6291456 tmpLen 262144 dataAddrP 0x38025E80000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483354265 72170 TRACE_KSVFS: cxiUXfer: nDesc 25 > currOffset 6553600 tmpLen 262144 dataAddrP 0x38025EC0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483766631 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > currOffset 6815744 tmpLen 262144 dataAddrP 0x38003B00000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483943894 72170 TRACE_KSVFS: cxiDetachIOBuffer: > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 > > 1.483944339 72170 TRACE_LOCK: unlock_vfs: type Data, > key 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode have ro > token xw lock_state old [ ro:14 ] new [ ro:13 ] holdCount now 14 > > 1.483944683 72170 TRACE_BRL: brUnlockM: ofP > 0xFFFFC90069346B68 inode 455025855 snap 0 handle 0xFFFFC9003637D020 range > 0x168F30D-0x268F30C mode ro > > 1.483944985 72170 TRACE_KSVFS: kSFSReadFast exit: > uio_resid 0 err 0 > > 1.483945264 72170 TRACE_LOCK: unlock_vfs_m: type > Inode, key 305F105B9701E60A:000000001B1F24BF:0000000000000000 lock_mode > have ro status valid token rs lock_state old [ ro:25 ] new [ ro:24 ] > > 1.483945423 72170 TRACE_LOCK: unlock_vfs_m: cP > 0xFFFFC90069346B68 holdCount 25 > > 1.483945624 72170 TRACE_VNODE: gpfsRead exit: fast err > 0 > > 1.483946831 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > 0xFFFFC90035E52F78 NotQuiesced vfsOp 2 > > 1.483946975 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > 0xFFFFC90035E52F78 vfsOp 2 users 1-1 > > 1.483947116 72170 TRACE_KSVFS: ReleaseDaemonSegAndSG: > sli 38 count 2 needCleanup 0 > > 1.483947593 72170 TRACE_VNODE: gpfs_f_rdwr exit: fP > 0xFFFF882541649400 total_len 16777216 uio_resid 0 offset 0x268F30D rc 0 > > > ??????????????????????????????????????????? > > > > Regards, > Lohit > > On Sep 19, 2018, 3:11 PM -0400, Sven Oehme , wrote: > > the document primarily explains all performance specific knobs. general > advice would be to longer set anything beside workerthreads, pagepool and > filecache on 5.X systems as most other settings are no longer relevant > (thats a client side statement) . thats is true until you hit strange > workloads , which is why all the knobs are still there :-) > > sven > > > On Wed, Sep 19, 2018 at 11:17 AM wrote: > >> Thanks Sven. >> I will disable it completely and see how it behaves. >> >> Is this the presentation? >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2014_UG10-5FGPFS-5FPerformance-5FSession-5Fv10.pdf&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=-Y1ncRDmgOCfWDQaXj-OcbiM5gcs0LcllAShfNapHtI&e= >> >> I guess i read it, but it did not strike me at this situation. I will try >> to read it again and see if i could make use of it. >> >> Regards, >> Lohit >> >> On Sep 19, 2018, 2:12 PM -0400, Sven Oehme , wrote: >> >> seem like you never read my performance presentation from a few years ago >> ;-) >> >> you can control this on a per node basis , either for all i/o : >> >> prefetchAggressiveness = X >> >> or individual for reads or writes : >> >> prefetchAggressivenessRead = X >> prefetchAggressivenessWrite = X >> >> for a start i would turn it off completely via : >> >> mmchconfig prefetchAggressiveness=0 -I -N nodename >> >> that will turn it off only for that node and only until you restart the >> node. >> then see what happens >> >> sven >> >> >> On Wed, Sep 19, 2018 at 11:07 AM wrote: >> >>> Thank you Sven. >>> >>> I mostly think it could be 1. or some other issue. >>> I don?t think it could be 2. , because i can replicate this issue no >>> matter what is the size of the dataset. It happens for few files that could >>> easily fit in the page pool too. >>> >>> I do see a lot more page faults for 16M compared to 1M, so it could be >>> related to many threads trying to compete for the same buffer space. >>> >>> I will try to take the trace with trace=io option and see if can find >>> something. >>> >>> How do i turn of prefetching? Can i turn it off for a single >>> node/client? >>> >>> Regards, >>> Lohit >>> >>> On Sep 18, 2018, 5:23 PM -0400, Sven Oehme , wrote: >>> >>> Hi, >>> >>> taking a trace would tell for sure, but i suspect what you might be >>> hitting one or even multiple issues which have similar negative performance >>> impacts but different root causes. >>> >>> 1. this could be serialization around buffer locks. as larger your >>> blocksize gets as larger is the amount of data one of this pagepool buffers >>> will maintain, if there is a lot of concurrency on smaller amount of data >>> more threads potentially compete for the same buffer lock to copy stuff in >>> and out of a particular buffer, hence things go slower compared to the same >>> amount of data spread across more buffers, each of smaller size. >>> >>> 2. your data set is small'ish, lets say a couple of time bigger than the >>> pagepool and you random access it with multiple threads. what will happen >>> is that because it doesn't fit into the cache it will be read from the >>> backend. if multiple threads hit the same 16 mb block at once with multiple >>> 4k random reads, it will read the whole 16mb block because it thinks it >>> will benefit from it later on out of cache, but because it fully random the >>> same happens with the next block and the next and so on and before you get >>> back to this block it was pushed out of the cache because of lack of enough >>> pagepool. >>> >>> i could think of multiple other scenarios , which is why its so hard to >>> accurately benchmark an application because you will design a benchmark to >>> test an application, but it actually almost always behaves different then >>> you think it does :-) >>> >>> so best is to run the real application and see under which configuration >>> it works best. >>> >>> you could also take a trace with trace=io and then look at >>> >>> TRACE_VNOP: READ: >>> TRACE_VNOP: WRITE: >>> >>> and compare them to >>> >>> TRACE_IO: QIO: read >>> TRACE_IO: QIO: write >>> >>> and see if the numbers summed up for both are somewhat equal. if >>> TRACE_VNOP is significant smaller than TRACE_IO you most likely do more i/o >>> than you should and turning prefetching off might actually make things >>> faster . >>> >>> keep in mind i am no longer working for IBM so all i say might be >>> obsolete by now, i no longer have access to the one and only truth aka the >>> source code ... but if i am wrong i am sure somebody will point this out >>> soon ;-) >>> >>> sven >>> >>> >>> >>> >>> On Tue, Sep 18, 2018 at 10:31 AM wrote: >>> >>>> Hello All, >>>> >>>> This is a continuation to the previous discussion that i had with Sven. >>>> However against what i had mentioned previously - i realize that this >>>> is ?not? related to mmap, and i see it when doing random freads. >>>> >>>> I see that block-size of the filesystem matters when reading from Page >>>> pool. >>>> I see a major difference in performance when compared 1M to 16M, when >>>> doing lot of random small freads with all of the data in pagepool. >>>> >>>> Performance for 1M is a magnitude ?more? than the performance that i >>>> see for 16M. >>>> >>>> The GPFS that we have currently is : >>>> Version : 5.0.1-0.5 >>>> Filesystem version: 19.01 (5.0.1.0) >>>> Block-size : 16M >>>> >>>> I had made the filesystem block-size to be 16M, thinking that i would >>>> get the most performance for both random/sequential reads from 16M than the >>>> smaller block-sizes. >>>> With GPFS 5.0, i made use the 1024 sub-blocks instead of 32 and thus >>>> not loose lot of storage space even with 16M. >>>> I had run few benchmarks and i did see that 16M was performing better >>>> ?when hitting storage/disks? with respect to bandwidth for >>>> random/sequential on small/large reads. >>>> >>>> However, with this particular workload - where it freads a chunk of >>>> data randomly from hundreds of files -> I see that the number of >>>> page-faults increase with block-size and actually reduce the performance. >>>> 1M performs a lot better than 16M, and may be i will get better >>>> performance with less than 1M. >>>> It gives the best performance when reading from local disk, with 4K >>>> block size filesystem. >>>> >>>> What i mean by performance when it comes to this workload - is not the >>>> bandwidth but the amount of time that it takes to do each iteration/read >>>> batch of data. >>>> >>>> I figure what is happening is: >>>> fread is trying to read a full block size of 16M - which is good in a >>>> way, when it hits the hard disk. >>>> But the application could be using just a small part of that 16M. Thus >>>> when randomly reading(freads) lot of data of 16M chunk size - it is page >>>> faulting a lot more and causing the performance to drop . >>>> I could try to make the application do read instead of freads, but i >>>> fear that could be bad too since it might be hitting the disk with a very >>>> small block size and that is not good. >>>> >>>> With the way i see things now - >>>> I believe it could be best if the application does random reads of >>>> 4k/1M from pagepool but some how does 16M from rotating disks. >>>> >>>> I don?t see any way of doing the above other than following a different >>>> approach where i create a filesystem with a smaller block size ( 1M or less >>>> than 1M ), on SSDs as a tier. >>>> >>>> May i please ask for advise, if what i am understanding/seeing is right >>>> and the best solution possible for the above scenario. >>>> >>>> Regards, >>>> Lohit >>>> >>>> On Apr 11, 2018, 10:36 AM -0400, Lohit Valleru , >>>> wrote: >>>> >>>> Hey Sven, >>>> >>>> This is regarding mmap issues and GPFS. >>>> We had discussed previously of experimenting with GPFS 5. >>>> >>>> I now have upgraded all of compute nodes and NSD nodes to GPFS 5.0.0.2 >>>> >>>> I am yet to experiment with mmap performance, but before that - I am >>>> seeing weird hangs with GPFS 5 and I think it could be related to mmap. >>>> >>>> Have you seen GPFS ever hang on this syscall? >>>> [Tue Apr 10 04:20:13 2018] [] >>>> _ZN10gpfsNode_t8mmapLockEiiPKj+0xb5/0x140 [mmfs26] >>>> >>>> I see the above ,when kernel hangs and throws out a series of trace >>>> calls. >>>> >>>> I somehow think the above trace is related to processes hanging on GPFS >>>> forever. There are no errors in GPFS however. >>>> >>>> Also, I think the above happens only when the mmap threads go above a >>>> particular number. >>>> >>>> We had faced a similar issue in 4.2.3 and it was resolved in a patch to >>>> 4.2.3.2 . At that time , the issue happened when mmap threads go more than >>>> worker1threads. According to the ticket - it was a mmap race condition that >>>> GPFS was not handling well. >>>> >>>> I am not sure if this issue is a repeat and I am yet to isolate the >>>> incident and test with increasing number of mmap threads. >>>> >>>> I am not 100 percent sure if this is related to mmap yet but just >>>> wanted to ask you if you have seen anything like above. >>>> >>>> Thanks, >>>> >>>> Lohit >>>> >>>> On Feb 22, 2018, 3:59 PM -0500, Sven Oehme , wrote: >>>> >>>> Hi Lohit, >>>> >>>> i am working with ray on a mmap performance improvement right now, >>>> which most likely has the same root cause as yours , see --> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_2018-2DJanuary_004411.html&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=AUEL847F_a-j6Y7t1fDMj4j33vLqvI6XrrNCVS5pUyA&e= >>>> the thread above is silent after a couple of back and rorth, but ray >>>> and i have active communication in the background and will repost as soon >>>> as there is something new to share. >>>> i am happy to look at this issue after we finish with ray's workload if >>>> there is something missing, but first let's finish his, get you try the >>>> same fix and see if there is something missing. >>>> >>>> btw. if people would share their use of MMAP , what applications they >>>> use (home grown, just use lmdb which uses mmap under the cover, etc) please >>>> let me know so i get a better picture on how wide the usage is with GPFS. i >>>> know a lot of the ML/DL workloads are using it, but i would like to know >>>> what else is out there i might not think about. feel free to drop me a >>>> personal note, i might not reply to it right away, but eventually. >>>> >>>> thx. sven >>>> >>>> >>>> On Thu, Feb 22, 2018 at 12:33 PM wrote: >>>> >>>>> Hi all, >>>>> >>>>> I wanted to know, how does mmap interact with GPFS pagepool with >>>>> respect to filesystem block-size? >>>>> Does the efficiency depend on the mmap read size and the block-size of >>>>> the filesystem even if all the data is cached in pagepool? >>>>> >>>>> GPFS 4.2.3.2 and CentOS7. >>>>> >>>>> Here is what i observed: >>>>> >>>>> I was testing a user script that uses mmap to read from 100M to 500MB >>>>> files. >>>>> >>>>> The above files are stored on 3 different filesystems. >>>>> >>>>> Compute nodes - 10G pagepool and 5G seqdiscardthreshold. >>>>> >>>>> 1. 4M block size GPFS filesystem, with separate metadata and data. >>>>> Data on Near line and metadata on SSDs >>>>> 2. 1M block size GPFS filesystem as a AFM cache cluster, "with all the >>>>> required files fully cached" from the above GPFS cluster as home. Data and >>>>> Metadata together on SSDs >>>>> 3. 16M block size GPFS filesystem, with separate metadata and data. >>>>> Data on Near line and metadata on SSDs >>>>> >>>>> When i run the script first time for ?each" filesystem: >>>>> I see that GPFS reads from the files, and caches into the pagepool as >>>>> it reads, from mmdiag -- iohist >>>>> >>>>> When i run the second time, i see that there are no IO requests from >>>>> the compute node to GPFS NSD servers, which is expected since all the data >>>>> from the 3 filesystems is cached. >>>>> >>>>> However - the time taken for the script to run for the files in the 3 >>>>> different filesystems is different - although i know that they are just >>>>> "mmapping"/reading from pagepool/cache and not from disk. >>>>> >>>>> Here is the difference in time, for IO just from pagepool: >>>>> >>>>> 20s 4M block size >>>>> 15s 1M block size >>>>> 40S 16M block size. >>>>> >>>>> Why do i see a difference when trying to mmap reads from different >>>>> block-size filesystems, although i see that the IO requests are not hitting >>>>> disks and just the pagepool? >>>>> >>>>> I am willing to share the strace output and mmdiag outputs if needed. >>>>> >>>>> Thanks, >>>>> Lohit >>>>> >>>>> _______________________________________________ >>>>> gpfsug-discuss mailing list >>>>> gpfsug-discuss at spectrumscale.org >>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= End of gpfsug-discuss Digest, Vol 81, Issue 43 ********************************************** ________________________________ This message is for the recipient's use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From valleru at cbio.mskcc.org Wed Nov 20 16:46:03 2019 From: valleru at cbio.mskcc.org (valleru at cbio.mskcc.org) Date: Wed, 20 Nov 2019 11:46:03 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 81, Issue 43 In-Reply-To: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> References: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> Message-ID: <2ff249fe-15d5-4d48-88fa-12ba40d1a0f8@Spark> Hey Tushit, For us, it finally came down to the kind of applications that are doing IO. For some applications, the 16M filesystem was very bad in performance, while for some it seems to do well. Yes, I do observe a performance difference with higher pagepool but the amount of pagepool also happens to depend on the working dataset of the applications. For us - Most of the compute nodes run a variety of applications with varying datasets. Thus modifying pagepool to a large value for all nodes did not seem to be practical/enough for us. I have not made prefetchaggressivenessread/large pagepool for every node yet. I am setting it only if it is making a big difference and only for dedicated compute nodes that run only specific set of applications. I believe the issue that I was facing with respect to large block size might be because of few things such as: 1. The amount of buffers/cache that can be held in a particular amount of pagepool. 2. If the application needs to read from the same buffers multiple times or from different buffers thus deciding the need for buffer locking issue as Sven mentioned or may be not enough buffers because of large block size and not enough pagepool memory, thus needing to page in/page out and read over the network to the storage. 3. The latency to read the full block size/16M from the storage and if the application is ok with this latency. I see that most of the GPU deep learning applications are not happy with large block size latency/IOPS/reads per second. 4. Most of the scientific applications including some libraries are designed to do buffered reads (full block size) from local disk and local disk mostly has ext4 4K block size so it should not affect much when done from local disk. But with GPFS 16M filesystem - they will read 16M from the GPFS filesystem/over the network ( though they don?t need all of the data in each 16M read). Thus with prefetch, and depending on IO pattern - GPFS can fetch lots of 16M reads unnecessarily needing for the OS to discard the rest of the data once it receives. In short - I had seen considerable difference when I had modified the applications to do unbuffered reads instead of buffered reads from a large block size filesystem. 5. The large block size reads also affected the Arista 100G network switch introducing lot of network discards, and the need for a higher buffer switch or enable ECN. 6. If the application does large sequential IO reads/large random IO reads or small random IO reads from many different files. The latter applications seems to be happy with smaller block size/larger pagepool/more maxfilestocache/network switch with lesser latency. While the former applications seem to do ok with large block size. Thus we have decided to debug/strace applications one a time, and see which applications are being affected by large block size and test them with a smaller block size on a SSD/Flash backed storage. Just yesterday, I was debugging one GPU deep learning application that does lot of small random IO from thousands of different files/latency sensitive and it performed best with small block size/large pagepool and more max files to cache/prefetchaggressivenessread to zero. The reason we had moved to 16M filesystem was only because of the reason that the rotating disk storage was most happy and gives maximum throughput when on larger block size ( ignoring the latency that this might introduce for each read). However we realized that not every application would be suited for a large block size, because of the above mentioned reasons. I had also seen that those applications are usually happy from local disk, because of the smaller block size that local disk offers and the dynamic filebuffer/cache that linux offers ( unlike pagepool which is static and pinned ). The filebuffer/cache has its advantages and disadvantages because of the varying performance depending on the free memory available. 4M block size might work better than 16M for such applications, but that will also reduce the total throughput that we get from the rotating disk storage - and not give much benefit with respect to latency/IOPS. So it all depends on the applications/IO pattern, and as of now - we are in process of gathering which applications are suited for large vs smaller block size. All the above is ignoring the metadata overhead/performance. Sometimes depending on the applications - metadata performance can be a bottleneck too. Regards, Lohit On Nov 20, 2019, 10:43 AM -0500, Tushit Shukla , wrote: > Lohit, > Did you get any progress on file system?performance issue with large block size ? > Apparently we are also going through the same issue after we migrated to 16M file system (from 4MB filesystem). Only thing which gives some relief is when we increase pagepool to 32G. We also tried playing with prefetchAggressivenessRead parameter by reducing it to 1 but not sure if it is doing anything. > > thanks > From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org > Sent: Monday, October 22, 2018 12:18:58 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 81, Issue 43 > > Send gpfsug-discuss mailing list submissions to > ??????? gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > ??????? https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > or, via email, send a message with subject or body 'help' to > ??????? gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > ??????? gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > ?? 1. Re: GPFS, Pagepool and Block size -> Perfomance reduces with > ????? larger block size (Sven Oehme) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 22 Oct 2018 09:18:43 -0700 > From: Sven Oehme > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS, Pagepool and Block size -> > ??????? Perfomance reduces with larger block size > Message-ID: > ??????? > Content-Type: text/plain; charset="utf-8" > > oops, somehow that slipped my inbox, i only saw that reply right now. > > its really hard to see from a trace snipped if the lock is the issue as the > lower level locks don't show up in default traces. without having access to > source code and a detailed trace you won't make much progress here. > > sven > > > > > > On Thu, Sep 27, 2018 at 12:31 PM wrote: > > > Thank you Sven, > > > > Turning of prefetching did not improve the performance, but it did degrade > > a bit. > > > > I have made the prefetching default and took trace dump, for tracectl with > > trace=io. Let me know if you want me to paste/attach it here. > > > > May i know, how could i confirm if the below is true? > > > > 1. this could be serialization around buffer locks. as larger your > >>> blocksize gets as larger is the amount of data one of this pagepool buffers > >>> will maintain, if there is a lot of concurrency on smaller amount of data > >>> more threads potentially compete for the same buffer lock to copy stuff in > >>> and out of a particular buffer, hence things go slower compared to the same > >>> amount of data spread across more buffers, each of smaller size. > >>> > >>> > > Will the above trace help in understanding if it is a serialization issue? > > > > I had been discussing the same with GPFS support for past few months, and > > it seems to be that most of the time is being spent at cxiUXfer. They could > > not understand on why it is taking spending so much of time in cxiUXfer. I > > was seeing the same from perf top, and pagefaults. > > > > Below is snippet from what the support had said : > > > > ???????????????????????????? > > > > I searched all of the gpfsRead from trace and sort them by spending-time. > > Except 2 reads which need fetch data from nsd server, the slowest read is > > in the thread 72170. It took 112470.362 us. > > > > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum:?? 72165?????? 6.860911319 > > rdwr?????????????????? 141857.076 us + NSDIO > > > > trcrpt.2018-08-06_12.26.28.39794.lt15.trsum:?? 72170?????? 1.483947593 > > rdwr?????????????????? 112470.362 us + cxiUXfer > > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum:?? 72165?????? 6.949042593 > > rdwr??????????????????? 88126.278 us + NSDIO > > > > trcrpt.2018-08-06_12.27.03.47706.lt15.trsum:?? 72156?????? 2.919334474 > > rdwr??????????????????? 81057.657 us + cxiUXfer > > > > trcrpt.2018-08-06_12.23.30.72745.lt15.trsum:?? 72154?????? 1.167484466 > > rdwr??????????????????? 76033.488 us + cxiUXfer > > > > trcrpt.2018-08-06_12.24.06.7508.lt15.trsum:?? 72187?????? 0.685237501 > > rdwr??????????????????? 70772.326 us + cxiUXfer > > > > trcrpt.2018-08-06_12.25.17.23989.lt15.trsum:?? 72193?????? 4.757996530 > > rdwr??????????????????? 70447.838 us + cxiUXfer > > > > > > I check each of the slow IO as above, and find they all spend much time in > > the function cxiUXfer. This function is used to copy data from kernel > > buffer to user buffer. I am not sure why it took so much time. This should > > be related to the pagefaults and pgfree you observed. Below is the trace > > data for thread 72170. > > > > > >??????????????????? 1.371477231? 72170 TRACE_VNODE: gpfs_f_rdwr enter: fP > > 0xFFFF882541649400 f_flags 0x8000 flags 0x8001 op 0 iovec > > 0xFFFF881F2AFB3E70 count 1 offset 0x168F30D dentry 0xFFFF887C0CC298C0 > > private 0xFFFF883F607175C0 iP 0xFFFF8823AA3CBFC0 name '410513.svs' > > > >?????????????? .... > > > >??????????????????? 1.371483547? 72170 TRACE_KSVFS: cachedReadFast exit: > > uio_resid 16777216 code 1 err 11 > > > >?????????????? .... > > > >??????????????????? 1.371498780? 72170 TRACE_KSVFS: kSFSReadFast: oiP > > 0xFFFFC90060B46740 offset 0x168F30D dataBufP FFFFC9003645A5A8 nDesc 64 buf > > 200043C0000 valid words 64 dirty words 0 blkOff 0 > > > >??????????????????? 1.371499035? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate begin ul 0xFFFFC900333F1A40 holdCount 0 > > ioType 0x2 inProg 0x15 > > > >??????????????????? 1.371500157? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate ul 0xFFFFC900333F1A40 holdCount 0 ioType 0x2 > > inProg 0x16 err 0 > > > >??????????????????? 1.371500606? 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 toIOBuf 0 offset 6877965 len > > 9899251 > > > >??????????????????? 1.371500793? 72170 TRACE_KSVFS: cxiUXfer: ndesc 0 skip > > dataAddrP 0x200043C0000 currOffset 0 currLen 262144 bufOffset 6877965 > > > >?????????????? .... > > > >??????????????????? 1.371505949? 72170 TRACE_KSVFS: cxiUXfer: ndesc 25 skip > > dataAddrP 0x2001AF80000 currOffset 6553600 currLen 262144 bufOffset 6877965 > > > >??????????????????? 1.371506236? 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > > currOffset 6815744 tmpLen 262144 dataAddrP 0x2001AFCF30D currLen 199923 > > pageOffset 781 pageLen 3315 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.373649823? 72170 TRACE_KSVFS: cxiUXfer: nDesc 27 > > currOffset 7077888 tmpLen 262144 dataAddrP 0x20027400000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.375158799? 72170 TRACE_KSVFS: cxiUXfer: nDesc 28 > > currOffset 7340032 tmpLen 262144 dataAddrP 0x20027440000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.376661566? 72170 TRACE_KSVFS: cxiUXfer: nDesc 29 > > currOffset 7602176 tmpLen 262144 dataAddrP 0x2002C180000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.377892653? 72170 TRACE_KSVFS: cxiUXfer: nDesc 30 > > currOffset 7864320 tmpLen 262144 dataAddrP 0x2002C1C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >?????????????? .... > > > >??????????????????? 1.471389843? 72170 TRACE_KSVFS: cxiUXfer: nDesc 62 > > currOffset 16252928 tmpLen 262144 dataAddrP 0x2001D2C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.471845629? 72170 TRACE_KSVFS: cxiUXfer: nDesc 63 > > currOffset 16515072 tmpLen 262144 dataAddrP 0x2003EC80000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.472417149? 72170 TRACE_KSVFS: cxiDetachIOBuffer: > > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.472417775? 72170 TRACE_LOCK: unlock_vfs: type Data, > > key 0000000000000004:000000001B1F24BF:0000000000000001 lock_mode have ro > > token xw lock_state old [ ro:27 ] new [ ro:26 ] holdCount now 27 > > > >??????????????????? 1.472418427? 72170 TRACE_LOCK: hash tab lookup vfs: > > found cP 0xFFFFC9005FC0CDE0 holdCount now 14 > > > >??????????????????? 1.472418592? 72170 TRACE_LOCK: lock_vfs: type Data key > > 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode want ro status > > valid token xw/xw lock_state [ ro:12 ] flags 0x0 holdCount 14 > > > >??????????????????? 1.472419842? 72170 TRACE_KSVFS: kSFSReadFast: oiP > > 0xFFFFC90060B46740 offset 0x2000000 dataBufP FFFFC9003643C908 nDesc 64 buf > > 38033480000 valid words 64 dirty words 0 blkOff 0 > > > >??????????????????? 1.472420029? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate begin ul 0xFFFFC9005FC0CF98 holdCount 0 > > ioType 0x2 inProg 0xC > > > >??????????????????? 1.472420187? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate ul 0xFFFFC9005FC0CF98 holdCount 0 ioType 0x2 > > inProg 0xD err 0 > > > >??????????????????? 1.472420652? 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 toIOBuf 0 offset 0 len 6877965 > > > >??????????????????? 1.472420936? 72170 TRACE_KSVFS: cxiUXfer: nDesc 0 > > currOffset 0 tmpLen 262144 dataAddrP 0x38033480000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.472824790? 72170 TRACE_KSVFS: cxiUXfer: nDesc 1 > > currOffset 262144 tmpLen 262144 dataAddrP 0x380334C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.473243905? 72170 TRACE_KSVFS: cxiUXfer: nDesc 2 > > currOffset 524288 tmpLen 262144 dataAddrP 0x38024280000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >?????????????? .... > > > >??????????????????? 1.482949347? 72170 TRACE_KSVFS: cxiUXfer: nDesc 24 > > currOffset 6291456 tmpLen 262144 dataAddrP 0x38025E80000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483354265? 72170 TRACE_KSVFS: cxiUXfer: nDesc 25 > > currOffset 6553600 tmpLen 262144 dataAddrP 0x38025EC0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483766631? 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > > currOffset 6815744 tmpLen 262144 dataAddrP 0x38003B00000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483943894? 72170 TRACE_KSVFS: cxiDetachIOBuffer: > > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483944339? 72170 TRACE_LOCK: unlock_vfs: type Data, > > key 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode have ro > > token xw lock_state old [ ro:14 ] new [ ro:13 ] holdCount now 14 > > > >??????????????????? 1.483944683? 72170 TRACE_BRL: brUnlockM: ofP > > 0xFFFFC90069346B68 inode 455025855 snap 0 handle 0xFFFFC9003637D020 range > > 0x168F30D-0x268F30C mode ro > > > >??????????????????? 1.483944985? 72170 TRACE_KSVFS: kSFSReadFast exit: > > uio_resid 0 err 0 > > > >??????????????????? 1.483945264? 72170 TRACE_LOCK: unlock_vfs_m: type > > Inode, key 305F105B9701E60A:000000001B1F24BF:0000000000000000 lock_mode > > have ro status valid token rs lock_state old [ ro:25 ] new [ ro:24 ] > > > >??????????????????? 1.483945423? 72170 TRACE_LOCK: unlock_vfs_m: cP > > 0xFFFFC90069346B68 holdCount 25 > > > >??????????????????? 1.483945624? 72170 TRACE_VNODE: gpfsRead exit: fast err > > 0 > > > >??????????????????? 1.483946831? 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > > 0xFFFFC90035E52F78 NotQuiesced vfsOp 2 > > > >??????????????????? 1.483946975? 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > > 0xFFFFC90035E52F78 vfsOp 2 users 1-1 > > > >??????????????????? 1.483947116? 72170 TRACE_KSVFS: ReleaseDaemonSegAndSG: > > sli 38 count 2 needCleanup 0 > > > >??????????????????? 1.483947593? 72170 TRACE_VNODE: gpfs_f_rdwr exit: fP > > 0xFFFF882541649400 total_len 16777216 uio_resid 0 offset 0x268F30D rc 0 > > > > > > ??????????????????????????????????????????? > > > > > > > > Regards, > > Lohit > > > > On Sep 19, 2018, 3:11 PM -0400, Sven Oehme , wrote: > > > > the document primarily explains all performance specific knobs. general > > advice would be to longer set anything beside workerthreads, pagepool and > > filecache on 5.X systems as most other settings are no longer relevant > > (thats a client side statement) . thats is true until you hit strange > > workloads , which is why all the knobs are still there :-) > > > > sven > > > > > > On Wed, Sep 19, 2018 at 11:17 AM wrote: > > > >> Thanks Sven. > >> I will disable it completely and see how it behaves. > >> > >> Is this the presentation? > >> > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2014_UG10-5FGPFS-5FPerformance-5FSession-5Fv10.pdf&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=-Y1ncRDmgOCfWDQaXj-OcbiM5gcs0LcllAShfNapHtI&e= > >> > >> I guess i read it, but it did not strike me at this situation. I will try > >> to read it again and see if i could make use of it. > >> > >> Regards, > >> Lohit > >> > >> On Sep 19, 2018, 2:12 PM -0400, Sven Oehme , wrote: > >> > >> seem like you never read my performance presentation from a few years ago > >> ;-) > >> > >> you can control this on a per node basis , either for all i/o : > >> > >>??? prefetchAggressiveness = X > >> > >> or individual for reads or writes : > >> > >>??? prefetchAggressivenessRead = X > >>??? prefetchAggressivenessWrite = X > >> > >> for a start i would turn it off completely via : > >> > >> mmchconfig prefetchAggressiveness=0 -I -N nodename > >> > >> that will turn it off only for that node and only until you restart the > >> node. > >> then see what happens > >> > >> sven > >> > >> > >> On Wed, Sep 19, 2018 at 11:07 AM wrote: > >> > >>> Thank you Sven. > >>> > >>> I mostly think it could be 1. or some other issue. > >>> I don?t think it could be 2. , because i can replicate this issue no > >>> matter what is the size of the dataset. It happens for few files that could > >>> easily fit in the page pool too. > >>> > >>> I do see a lot more page faults for 16M compared to 1M, so it could be > >>> related to many threads trying to compete for the same buffer space. > >>> > >>> I will try to take the trace with trace=io option and see if can find > >>> something. > >>> > >>> How do i turn of prefetching? Can i turn it off for a single > >>> node/client? > >>> > >>> Regards, > >>> Lohit > >>> > >>> On Sep 18, 2018, 5:23 PM -0400, Sven Oehme , wrote: > >>> > >>> Hi, > >>> > >>> taking a trace would tell for sure, but i suspect what you might be > >>> hitting one or even multiple issues which have similar negative performance > >>> impacts but different root causes. > >>> > >>> 1. this could be serialization around buffer locks. as larger your > >>> blocksize gets as larger is the amount of data one of this pagepool buffers > >>> will maintain, if there is a lot of concurrency on smaller amount of data > >>> more threads potentially compete for the same buffer lock to copy stuff in > >>> and out of a particular buffer, hence things go slower compared to the same > >>> amount of data spread across more buffers, each of smaller size. > >>> > >>> 2. your data set is small'ish, lets say a couple of time bigger than the > >>> pagepool and you random access it with multiple threads. what will happen > >>> is that because it doesn't fit into the cache it will be read from the > >>> backend. if multiple threads hit the same 16 mb block at once with multiple > >>> 4k random reads, it will read the whole 16mb block because it thinks it > >>> will benefit from it later on out of cache, but because it fully random the > >>> same happens with the next block and the next and so on and before you get > >>> back to this block it was pushed out of the cache because of lack of enough > >>> pagepool. > >>> > >>> i could think of multiple other scenarios , which is why its so hard to > >>> accurately benchmark an application because you will design a benchmark to > >>> test an application, but it actually almost always behaves different then > >>> you think it does :-) > >>> > >>> so best is to run the real application and see under which configuration > >>> it works best. > >>> > >>> you could also take a trace with trace=io and then look at > >>> > >>> TRACE_VNOP: READ: > >>> TRACE_VNOP: WRITE: > >>> > >>> and compare them to > >>> > >>> TRACE_IO: QIO: read > >>> TRACE_IO: QIO: write > >>> > >>> and see if the numbers summed up for both are somewhat equal. if > >>> TRACE_VNOP is significant smaller than TRACE_IO you most likely do more i/o > >>> than you should and turning prefetching off might actually make things > >>> faster . > >>> > >>> keep in mind i am no longer working for IBM so all i say might be > >>> obsolete by now, i no longer have access to the one and only truth aka the > >>> source code ... but if i am wrong i am sure somebody will point this out > >>> soon ;-) > >>> > >>> sven > >>> > >>> > >>> > >>> > >>> On Tue, Sep 18, 2018 at 10:31 AM wrote: > >>> > >>>> Hello All, > >>>> > >>>> This is a continuation to the previous discussion that i had with Sven. > >>>> However against what i had mentioned previously - i realize that this > >>>> is ?not? related to mmap, and i see it when doing random freads. > >>>> > >>>> I see that block-size of the filesystem matters when reading from Page > >>>> pool. > >>>> I see a major difference in performance when compared 1M to 16M, when > >>>> doing lot of random small freads with all of the data in pagepool. > >>>> > >>>> Performance for 1M is a magnitude ?more? than the performance that i > >>>> see for 16M. > >>>> > >>>> The GPFS that we have currently is : > >>>> Version : 5.0.1-0.5 > >>>> Filesystem version: 19.01 (5.0.1.0) > >>>> Block-size : 16M > >>>> > >>>> I had made the filesystem block-size to be 16M, thinking that i would > >>>> get the most performance for both random/sequential reads from 16M than the > >>>> smaller block-sizes. > >>>> With GPFS 5.0, i made use the 1024 sub-blocks instead of 32 and thus > >>>> not loose lot of storage space even with 16M. > >>>> I had run few benchmarks and i did see that 16M was performing better > >>>> ?when hitting storage/disks? with respect to bandwidth for > >>>> random/sequential on small/large reads. > >>>> > >>>> However, with this particular workload - where it freads a chunk of > >>>> data randomly from hundreds of files -> I see that the number of > >>>> page-faults increase with block-size and actually reduce the performance. > >>>> 1M performs a lot better than 16M, and may be i will get better > >>>> performance with less than 1M. > >>>> It gives the best performance when reading from local disk, with 4K > >>>> block size filesystem. > >>>> > >>>> What i mean by performance when it comes to this workload - is not the > >>>> bandwidth but the amount of time that it takes to do each iteration/read > >>>> batch of data. > >>>> > >>>> I figure what is happening is: > >>>> fread is trying to read a full block size of 16M - which is good in a > >>>> way, when it hits the hard disk. > >>>> But the application could be using just a small part of that 16M. Thus > >>>> when randomly reading(freads) lot of data of 16M chunk size - it is page > >>>> faulting a lot more and causing the performance to drop . > >>>> I could try to make the application do read instead of freads, but i > >>>> fear that could be bad too since it might be hitting the disk with a very > >>>> small block size and that is not good. > >>>> > >>>> With the way i see things now - > >>>> I believe it could be best if the application does random reads of > >>>> 4k/1M from pagepool but some how does 16M from rotating disks. > >>>> > >>>> I don?t see any way of doing the above other than following a different > >>>> approach where i create a filesystem with a smaller block size ( 1M or less > >>>> than 1M ), on SSDs as a tier. > >>>> > >>>> May i please ask for advise, if what i am understanding/seeing is right > >>>> and the best solution possible for the above scenario. > >>>> > >>>> Regards, > >>>> Lohit > >>>> > >>>> On Apr 11, 2018, 10:36 AM -0400, Lohit Valleru , > >>>> wrote: > >>>> > >>>> Hey Sven, > >>>> > >>>> This is regarding mmap issues and GPFS. > >>>> We had discussed previously of experimenting with GPFS 5. > >>>> > >>>> I now have upgraded all of compute nodes and NSD nodes to GPFS 5.0.0.2 > >>>> > >>>> I am yet to experiment with mmap performance, but before that - I am > >>>> seeing weird hangs with GPFS 5 and I think it could be related to mmap. > >>>> > >>>> Have you seen GPFS ever hang on this syscall? > >>>> [Tue Apr 10 04:20:13 2018] [] > >>>> _ZN10gpfsNode_t8mmapLockEiiPKj+0xb5/0x140 [mmfs26] > >>>> > >>>> I see the above ,when kernel hangs and throws out a series of trace > >>>> calls. > >>>> > >>>> I somehow think the above trace is related to processes hanging on GPFS > >>>> forever. There are no errors in GPFS however. > >>>> > >>>> Also, I think the above happens only when the mmap threads go above a > >>>> particular number. > >>>> > >>>> We had faced a similar issue in 4.2.3 and it was resolved in a patch to > >>>> 4.2.3.2 . At that time , the issue happened when mmap threads go more than > >>>> worker1threads. According to the ticket - it was a mmap race condition that > >>>> GPFS was not handling well. > >>>> > >>>> I am not sure if this issue is a repeat and I am yet to isolate the > >>>> incident and test with increasing number of mmap threads. > >>>> > >>>> I am not 100 percent sure if this is related to mmap yet but just > >>>> wanted to ask you if you have seen anything like above. > >>>> > >>>> Thanks, > >>>> > >>>> Lohit > >>>> > >>>> On Feb 22, 2018, 3:59 PM -0500, Sven Oehme , wrote: > >>>> > >>>> Hi Lohit, > >>>> > >>>> i am working with ray on a mmap performance improvement right now, > >>>> which most likely has the same root cause as yours , see --> > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_2018-2DJanuary_004411.html&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=AUEL847F_a-j6Y7t1fDMj4j33vLqvI6XrrNCVS5pUyA&e= > >>>> the thread above is silent after a couple of back and rorth, but ray > >>>> and i have active communication in the background and will repost as soon > >>>> as there is something new to share. > >>>> i am happy to look at this issue after we finish with ray's workload if > >>>> there is something missing, but first let's finish his, get you try the > >>>> same fix and see if there is something missing. > >>>> > >>>> btw. if people would share their use of MMAP , what applications they > >>>> use (home grown, just use lmdb which uses mmap under the cover, etc) please > >>>> let me know so i get a better picture on how wide the usage is with GPFS. i > >>>> know a lot of the ML/DL workloads are using it, but i would like to know > >>>> what else is out there i might not think about. feel free to drop me a > >>>> personal note, i might not reply to it right away, but eventually. > >>>> > >>>> thx. sven > >>>> > >>>> > >>>> On Thu, Feb 22, 2018 at 12:33 PM wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> I wanted to know, how does mmap interact with GPFS pagepool with > >>>>> respect to filesystem block-size? > >>>>> Does the efficiency depend on the mmap read size and the block-size of > >>>>> the filesystem even if all the data is cached in pagepool? > >>>>> > >>>>> GPFS 4.2.3.2 and CentOS7. > >>>>> > >>>>> Here is what i observed: > >>>>> > >>>>> I was testing a user script that uses mmap to read from 100M to 500MB > >>>>> files. > >>>>> > >>>>> The above files are stored on 3 different filesystems. > >>>>> > >>>>> Compute nodes - 10G pagepool and 5G seqdiscardthreshold. > >>>>> > >>>>> 1. 4M block size GPFS filesystem, with separate metadata and data. > >>>>> Data on Near line and metadata on SSDs > >>>>> 2. 1M block size GPFS filesystem as a AFM cache cluster, "with all the > >>>>> required files fully cached" from the above GPFS cluster as home. Data and > >>>>> Metadata together on SSDs > >>>>> 3. 16M block size GPFS filesystem, with separate metadata and data. > >>>>> Data on Near line and metadata on SSDs > >>>>> > >>>>> When i run the script first time for ?each" filesystem: > >>>>> I see that GPFS reads from the files, and caches into the pagepool as > >>>>> it reads, from mmdiag -- iohist > >>>>> > >>>>> When i run the second time, i see that there are no IO requests from > >>>>> the compute node to GPFS NSD servers, which is expected since all the data > >>>>> from the 3 filesystems is cached. > >>>>> > >>>>> However - the time taken for the script to run for the files in the 3 > >>>>> different filesystems is different - although i know that they are just > >>>>> "mmapping"/reading from pagepool/cache and not from disk. > >>>>> > >>>>> Here is the difference in time, for IO just from pagepool: > >>>>> > >>>>> 20s 4M block size > >>>>> 15s 1M block size > >>>>> 40S 16M block size. > >>>>> > >>>>> Why do i see a difference when trying to mmap reads from different > >>>>> block-size filesystems, although i see that the IO requests are not hitting > >>>>> disks and just the pagepool? > >>>>> > >>>>> I am willing to share the strace output and mmdiag outputs if needed. > >>>>> > >>>>> Thanks, > >>>>> Lohit > >>>>> > >>>>> _______________________________________________ > >>>>> gpfsug-discuss mailing list > >>>>> gpfsug-discuss at spectrumscale.org > >>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >> > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > End of gpfsug-discuss Digest, Vol 81, Issue 43 > ********************************************** > This message is for the recipient?s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From WPeters at ATPCO.NET Wed Nov 20 18:16:23 2019 From: WPeters at ATPCO.NET (Peters, Bill) Date: Wed, 20 Nov 2019 18:16:23 +0000 Subject: [gpfsug-discuss] introduction Message-ID: Hello, The welcome email said I should introduce myself to the group. I'm Bill Peters, a Linux engineer at ATPCO (Airline Tariff Publishing Company) located in Dulles, Virginia, USA. We process airline fare data. I've was recently introduced to Spectrum Scale because we are doing a proof of concept to see if we can move some of our workload onto virtual machines running in IBM's zVM. Our x86 systems use Veritas for the network filesystem and since Veritas doesn't support the s390 arcitecture, we are using Spectrum Scale. So far it's been great, much easier to understand than Veritas. We're not doing anything too complicated. The disks are DASD on SSD. We have 3 clusters with sharing between them. At this point I expect the POC to be successful so I will probably be working with Spectrum Scale into the future. The only question I have so far is read speeds seem to be slower than Veritas. It's not slow enough to be a real problem, but if anyone has a suggestion for speeding up reads, I'd love to hear it. Other than that, I'll probably just be lurking on the email list. Thanks, -Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Nov 20 19:56:50 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 20 Nov 2019 20:56:50 +0100 Subject: [gpfsug-discuss] introduction In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Nov 20 20:06:53 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 20 Nov 2019 21:06:53 +0100 Subject: [gpfsug-discuss] introduction In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Nov 20 20:44:05 2019 From: mweil at wustl.edu (Weil, Matthew) Date: Wed, 20 Nov 2019 20:44:05 +0000 Subject: [gpfsug-discuss] fsstruct and FSCK IBM case TS002836092 Message-ID: <5889a4ef-0c62-8ee8-7347-a7bb6b3acb74@wustl.edu> Ulf Troppens, Would it be possible for you to engage and assist with this case? Thanks Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From gterryc at vmsupport.com Fri Nov 22 00:24:02 2019 From: gterryc at vmsupport.com (George Terry) Date: Thu, 21 Nov 2019 18:24:02 -0600 Subject: [gpfsug-discuss] Compatibility question Message-ID: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at googlemail.com Fri Nov 22 01:12:27 2019 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Thu, 21 Nov 2019 19:12:27 -0600 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: Message-ID: That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, wrote: > Hello, > > I have a question about compatibility between two different versions. I > have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are > clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to > application's issues I cannot update the client nodes. So, my question is, > can i have a cluster with client nodes with 3.5.0.7 version and the NSD > nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? > > Thanks for the support. > > George > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knop at us.ibm.com Fri Nov 22 04:49:12 2019 From: knop at us.ibm.com (Felipe Knop) Date: Fri, 22 Nov 2019 04:49:12 +0000 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Fri Nov 22 14:10:01 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Fri, 22 Nov 2019 14:10:01 +0000 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: Message-ID: My guess would be rhel6 clients. Though rhel6 should be able to run a 4.x release and then you are N-1. Our one and only remaining centos6 box, well we unceremoniously kicked that out of the cluster and made it use NFS. Simon From: on behalf of "luke.raimbach at googlemail.com" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 22 November 2019 at 01:12 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Compatibility question That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, > wrote: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From sadaniel at us.ibm.com Fri Nov 22 17:01:09 2019 From: sadaniel at us.ibm.com (Steven Daniels) Date: Fri, 22 Nov 2019 10:01:09 -0700 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: , Message-ID: Felipe, You are correct. The code is restricted to 4.2.2 or higher. If there is an ESS storage cluster in the environment then the recommendation I was told is 4.2.3.7 or higher. Luke, If your clients are in the same cluster then you need to be careful with the setting to minReleaseLevel, Exactly as it sounds, it will restrict membership to this level or higher. I have a client running with filesystem version 3.5.0.7 in several clusters. We just recently upgraded their ESS systems to the current release so I can verify that the file system format will carry fine. If you have ESS at 5.3.3 or higher, then the GNR recommendation is 4.2.3.7 or higher for the clients (my client can't go to v5 due to the RHEL version requirement) and they must be in a remote cluster. In my case we upgraded an ESS storage cluster to 4.1.1 to 5.0.3.3 (ESS v5.3.4.1) and for GNR we had to update this cluster to LATEST for minReleaseLevel which is 5.0.3. The client cluster is running 4.2.3.18 which is as Felipe says is N-1 and minReleaseLevel is set to 4.2.3.7 This client cluster also mounts from another storage cluster which is constrained to run at 4.1.1-8. This makes the client cluster at N+1 from its relationship with this cluster. All three clusters have minReleaseLevel set to LATEST (4.1.0, 4.2.3.7,5.0.3 , respectively). These changes would not work in a single cluster as all nodes would be restricted to the 4.1.0 or higher and the ESS would not be able to co-exist since it requires the 5.0 level. We have updated the file systems in the 4.1.1-8 cluster from 3.5.0.7 to 4.1.1 and we updated the ESS filesystems from 3.5.0.7 to 5.0.3 (compat mode) The changes all went well. Hopefully this helps you understand the nuance that is whether the clients and NSDs are in a single cluster or different clusters. The rules are slightly but importantly different. Also, when updating the filesystem versions make sure you use the "compat" flag if you have any remote clusters. If we would have run this with the "full" flag on the ESS storage cluster, it would have prevented the 4.2.3.18 client cluster from mounting its file systems. thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Felipe Knop" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 11/21/2019 09:49 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question Sent by: gpfsug-discuss-bounces at spectrumscale.org George, Coexistence either in the same cluster or across remote clusters is only supported with versions "N" and "N-1". 5.0.3 and 3.5 are "N" and "N-3", and this is not supported. If I recall, there is even logic in place that will prevent 5.x nodes from accepting connections from nodes running below 4.2 . Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 ----- Original message ----- From: Luke Raimbach Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question Date: Thu, Nov 21, 2019 8:12 PM That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, wrote: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=sHhNGNTp4CJIVbXT4t8ZvNT_mETKfXBn7XTvNDAYF7s&s=9ialbzpuZMc8DZzXtU_nqjgQ9BPsHP3DQdpHa0nKQuc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: From werner at usit.uio.no Tue Nov 26 06:25:08 2019 From: werner at usit.uio.no (Morten Werner Forsbring) Date: Tue, 26 Nov 2019 07:25:08 +0100 Subject: [gpfsug-discuss] New subscriber Message-ID: <87blsz6tob.fsf@sue.uio.no> Hi. My name is Morten Werner Forsbring and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with different services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Best regards! - Werner From helge.hauglin at usit.uio.no Tue Nov 26 10:25:21 2019 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 26 Nov 2019 11:25:21 +0100 Subject: [gpfsug-discuss] New member: Helge Hauglin Message-ID: Hi, My name is Helge Hauglin, and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Previous storage experience: NetApp and Hitachi Vantara (Hitachi NAS and block storage). Best regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From christophe.darras at atempo.com Tue Nov 26 11:21:29 2019 From: christophe.darras at atempo.com (Christophe Darras) Date: Tue, 26 Nov 2019 11:21:29 +0000 Subject: [gpfsug-discuss] New member: Helge Hauglin In-Reply-To: References: Message-ID: Hello Helge, Thanks for the introduction. Might you need some help to migrate your unstructured data from HDS and NetApp to GPFS, pls let us know, Kindest Regards, Christophe DARRAS ? Head of North Europe, Mobile : +44 7?555 99 3529 christophe.darras at atempo.com Atempo ltd N?1 EUROPEAN SOFTWARE VENDOR FOR DATA PROTECTION? ? ATEMPO.COM ? ? ? ? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Helge Hauglin Sent: mardi 26 novembre 2019 10:25 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] New member: Helge Hauglin Hi, My name is Helge Hauglin, and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Previous storage experience: NetApp and Hitachi Vantara (Hitachi NAS and block storage). Best regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From helge.hauglin at usit.uio.no Tue Nov 26 13:21:10 2019 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 26 Nov 2019 14:21:10 +0100 Subject: [gpfsug-discuss] New member: Helge Hauglin In-Reply-To: (Christophe Darras's message of "Tue, 26 Nov 2019 11:21:29 +0000") References: Message-ID: Hi Christophe, Thanks for the information, but we'll use our in-house expertise to move data to the ESS. -- Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From mark.bergman at uphs.upenn.edu Tue Nov 26 23:58:02 2019 From: mark.bergman at uphs.upenn.edu (mark.bergman at uphs.upenn.edu) Date: Tue, 26 Nov 2019 18:58:02 -0500 Subject: [gpfsug-discuss] python package installs fail due to attribute copy error (GPFS 5.0.2) Message-ID: <32070-1574812682.257155@ttRP.UhP3.tHtF> We're running GPFS 5.0.2 (DSS-G 2.2a, in the process of upgrading to 2.4b/GPFS 5.0.3.2) with NFSv4 ACLs and no explicit ACLs on the root or dependent filesets. While installing python packages within a python virtual env stored under GPFS, we get failures with "permission denied" on the destination while trying to copy attributes. The problem happens to root & end-users, and is not related to the directory permissions, ownership, or group, and is consistent and repeatable. For example, trying to use 'conda' to create a virtual environment results in the following message (severely trimmed): # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): in clone_env shutil.copystat(src, dst) in copystat _copyxattr(src, dst, follow_symlinks=follow) in _copyxattr os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) PermissionError: [Errno 13] Permission denied: '/gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6/lib/python3.5/site-packages/dm/neuralnet/__pycache__/__init__.cpython-35.pyc' `$ python/anaconda/3/bin/conda create --prefix /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6 --clone /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6` Note that the destination directory path is created, and the destination file (__init__.cython-35.pyc) is created...the exception is thrown when python tries to apply the ACLs from the source to the destination. The problem seems to be simiar to this: https://github.com/conda/conda/issues/5251 but our version of create.py is newer and already has the suggested fix. or similar to this: https://bugs.python.org/issue24538 that's affecting panfs. We've got a work-around using symlinks, but it's not something for average users. Has anyone seen this issue? Thanks, Mark "about to post to various python fora" Bergman From b.cregan at imperial.ac.uk Thu Nov 28 09:43:55 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 09:43:55 +0000 Subject: [gpfsug-discuss] Compression question Message-ID: Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 09:49:32 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 09:49:32 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Thu Nov 28 10:05:23 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 10:05:23 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 10:13:35 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 10:13:35 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 12:21:00 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 12:21:00 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Thu Nov 28 12:30:25 2019 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 28 Nov 2019 12:30:25 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 12:49:01 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 12:49:01 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191283.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191284.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191285.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Thu Nov 28 12:57:37 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 12:57:37 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , , Message-ID: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com [https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png] [https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png] [https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png] ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards [cid:15749424191280] [IBM Spectrum Scale] * * * Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com [cid:15749424191282] IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: Image.15749424191280.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: Image.15749424191281.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: Image.15749424191282.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 13:00:22 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 13:00:22 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: Message-ID: Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers > On 28. Nov 2019, at 14.57, Cregan, Bob wrote: > > ? > Hi > Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? > > After compression of an existing file we get in the snap > > -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf > file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf > metadata replication: 2 max 2 > data replication: 1 max 3 > immutable: no > appendOnly: no > flags: > storage pool name: sata1 > fileset name: userdirs > snapshot name: @GMT-2019.11.27-19.30.14 > creation time: Tue Mar 5 16:16:40 2019 > Misc attributes: ARCHIVE > Encrypted: no > > and the original file is definitely compressed. > > -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf > file name: UserGuide_13.06.pdf > metadata replication: 2 max 2 > data replication: 1 max 3 > immutable: no > appendOnly: no > flags: > storage pool name: sata1 > fileset name: userdirs > snapshot name: > creation time: Tue Mar 5 16:16:40 2019 > Misc attributes: ARCHIVE COMPRESSION (library z) > Encrypted: no > > Bob > > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger > Sent: 28 November 2019 12:30 > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Compression question > > Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial > > > Alexander, > > Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? > Daniel > > _________________________________________________________ > Daniel Kidger > IBM Technical Sales Specialist > Spectrum Scale, Spectrum NAS and IBM Cloud Object Store > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > > > > ----- Original message ----- > From: "Alexander Wolf" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 12:21 > > I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). > > Mit freundlichen Gr??en / Kind regards > > > > > > Dr. Alexander Wolf-Reber > Spectrum Scale Release Lead Architect > Department M069 / Spectrum Scale Software Development > > +49-160-90540880 > a.wolf-reber at de.ibm.com > > IBM Data Privacy Statement > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > ----- Original message ----- > From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 11:47 > > Hi > > Same principle COW. The data blocks do not get modified. > -- > Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions > Luis Bolinches > Consultant IT Specialist > Mobile Phone: +358503112585 > https://www.youracclaim.com/user/luis-bolinches > > "If you always give you will always have" -- Anonymous > > > > ----- Original message ----- > From: "Cregan, Bob" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 12:06 PM > > Just to clarify this is SS compression, so > > mmchattr --compression yes > > or an ILM equivalent > > So not a regular modification. > > Bob > > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > > > > From: Cregan, Bob > Sent: 28 November 2019 09:43 > To: gpfsug main discussion list > Subject: Compression question > > Hi All, > Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. > > What happens to the snapshot when a file is compressed in SS? The logic as I see it is > > ####### In a non compressed situation ############### > > 1) create a file, > 2) create a snapshot. > 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. > > ######In a compressed situation ############ > > 1) create a file, > 2) create a snapshot. > 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. > > You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. > Thanks > > Bob > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.cregan at imperial.ac.uk Thu Nov 28 13:05:43 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 13:05:43 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: Hi 5.0.2 with a hotfix for a lz4 compression bug Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches Sent: 28 November 2019 13:00 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Compression question Caution - This email from luis.bolinches at fi.ibm.com originated outside Imperial Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob wrote: ? Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com [https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png] [https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png] [https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png] ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards * * * Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 14:02:46 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 14:02:46 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191286.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191287.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191288.png Type: image/png Size: 1134 bytes Desc: not available URL: From luis.bolinches at fi.ibm.com Thu Nov 28 14:07:27 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 14:07:27 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: Message-ID: That is the fix. Before it was an error. So blocks that are pointed from active get compressed. Ones that are no longer linked are not modified. -- Cheers > On 28. Nov 2019, at 16.03, Alexander Wolf wrote: > > ? > I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. > > Mit freundlichen Gr??en / Kind regards > > > > > > Dr. Alexander Wolf-Reber > Spectrum Scale Release Lead Architect > Department M069 / Spectrum Scale Software Development > > +49-160-90540880 > a.wolf-reber at de.ibm.com > > IBM Data Privacy Statement > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > ----- Original message ----- > From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: "gpfsug main discussion list" > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 14:00 > > Which version are you running? > > I was involved on a big for compressed file sets and snapshots that were related to what you see. > > -- > Cheers > >> >>> On 28. Nov 2019, at 14.57, Cregan, Bob wrote: >>> >> ? >> Hi >> Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? >> >> After compression of an existing file we get in the snap >> >> -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf >> file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf >> metadata replication: 2 max 2 >> data replication: 1 max 3 >> immutable: no >> appendOnly: no >> flags: >> storage pool name: sata1 >> fileset name: userdirs >> snapshot name: @GMT-2019.11.27-19.30.14 >> creation time: Tue Mar 5 16:16:40 2019 >> Misc attributes: ARCHIVE >> Encrypted: no >> >> and the original file is definitely compressed. >> >> -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf >> file name: UserGuide_13.06.pdf >> metadata replication: 2 max 2 >> data replication: 1 max 3 >> immutable: no >> appendOnly: no >> flags: >> storage pool name: sata1 >> fileset name: userdirs >> snapshot name: >> creation time: Tue Mar 5 16:16:40 2019 >> Misc attributes: ARCHIVE COMPRESSION (library z) >> Encrypted: no >> Bob >> >> >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger >> Sent: 28 November 2019 12:30 >> To: gpfsug-discuss at spectrumscale.org >> Cc: gpfsug-discuss at spectrumscale.org >> Subject: Re: [gpfsug-discuss] Compression question >> >> Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial >> >> >> Alexander, >> >> Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? >> Daniel >> >> _________________________________________________________ >> Daniel Kidger >> IBM Technical Sales Specialist >> Spectrum Scale, Spectrum NAS and IBM Cloud Object Store >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> >> >> >> ----- Original message ----- >> From: "Alexander Wolf" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 12:21 >> >> I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). >> >> Mit freundlichen Gr??en / Kind regards >> >> >> >> >> >> >> >> >> Dr. Alexander Wolf-Reber >> Spectrum Scale Release Lead Architect >> Department M069 / Spectrum Scale Software Development >> >> +49-160-90540880 >> a.wolf-reber at de.ibm.com >> >> IBM Data Privacy Statement >> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp >> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 >> >> >> >> ----- Original message ----- >> From: "Luis Bolinches" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: gpfsug-discuss at spectrumscale.org >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 11:47 >> >> Hi >> >> Same principle COW. The data blocks do not get modified. >> -- >> Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions >> Luis Bolinches >> Consultant IT Specialist >> Mobile Phone: +358503112585 >> https://www.youracclaim.com/user/luis-bolinches >> >> "If you always give you will always have" -- Anonymous >> >> >> >> ----- Original message ----- >> From: "Cregan, Bob" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 12:06 PM >> >> Just to clarify this is SS compression, so >> >> mmchattr --compression yes >> >> or an ILM equivalent >> >> So not a regular modification. >> >> Bob >> >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> >> From: Cregan, Bob >> Sent: 28 November 2019 09:43 >> To: gpfsug main discussion list >> Subject: Compression question >> >> Hi All, >> Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. >> >> What happens to the snapshot when a file is compressed in SS? The logic as I see it is >> >> ####### In a non compressed situation ############### >> >> 1) create a file, >> 2) create a snapshot. >> 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. >> >> ######In a compressed situation ############ >> >> 1) create a file, >> 2) create a snapshot. >> 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. >> >> You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. >> Thanks >> >> Bob >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Thu Nov 28 15:00:56 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 28 Nov 2019 08:00:56 -0700 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 1134 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Thu Nov 28 15:28:28 2019 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 28 Nov 2019 15:28:28 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB17BD4DEB177B80052596E872584C0.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB18B14DEB187280052596E872584C0.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB19CD4DEB180B40052596E872584C0.png Type: image/png Size: 1134 bytes Desc: not available URL: From scale at us.ibm.com Thu Nov 28 17:09:37 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 28 Nov 2019 22:39:37 +0530 Subject: [gpfsug-discuss] =?utf-8?q?python_package_installs_fail_due_to_at?= =?utf-8?q?tribute_copy=09error_=28GPFS_5=2E0=2E2=29?= In-Reply-To: <32070-1574812682.257155@ttRP.UhP3.tHtF> References: <32070-1574812682.257155@ttRP.UhP3.tHtF> Message-ID: Hi Diane, Can you please help customer with the below issue. Or else can you point me to the right folks who can help here. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: mark.bergman at uphs.upenn.edu To: gpfsug main discussion list Date: 27-11-2019 05:47 Subject: [EXTERNAL] [gpfsug-discuss] python package installs fail due to attribute copy error (GPFS 5.0.2) Sent by: gpfsug-discuss-bounces at spectrumscale.org We're running GPFS 5.0.2 (DSS-G 2.2a, in the process of upgrading to 2.4b/GPFS 5.0.3.2) with NFSv4 ACLs and no explicit ACLs on the root or dependent filesets. While installing python packages within a python virtual env stored under GPFS, we get failures with "permission denied" on the destination while trying to copy attributes. The problem happens to root & end-users, and is not related to the directory permissions, ownership, or group, and is consistent and repeatable. For example, trying to use 'conda' to create a virtual environment results in the following message (severely trimmed): # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): in clone_env shutil.copystat(src, dst) in copystat _copyxattr(src, dst, follow_symlinks=follow) in _copyxattr os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) PermissionError: [Errno 13] Permission denied: '/gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6/lib/python3.5/site-packages/dm/neuralnet/__pycache__/__init__.cpython-35.pyc' `$ python/anaconda/3/bin/conda create --prefix /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6 --clone /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6` Note that the destination directory path is created, and the destination file (__init__.cython-35.pyc) is created...the exception is thrown when python tries to apply the ACLs from the source to the destination. The problem seems to be simiar to this: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda_conda_issues_5251&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=8cqECGE7aQzVZwSOUDWjUbgMiyWLzpVqThnCrhMXwGc&e= but our version of create.py is newer and already has the suggested fix. or similar to this: https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue24538&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=Zyfy_KMaSuhXSmm5sUTBMiRqEiaIfD4BLSyRfpNjQkY&e= that's affecting panfs. We've got a work-around using symlinks, but it's not something for average users. Has anyone seen this issue? Thanks, Mark "about to post to various python fora" Bergman _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=uVXOGz3UqVUU4d3Ov4e4-8wLRnxTsNtRPOv_iWcmOjM&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From scale at us.ibm.com Thu Nov 28 17:22:43 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 28 Nov 2019 22:52:43 +0530 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: Hai Zhong, Can you please help the customer with their compression related query. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Daniel Kidger" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 28-11-2019 20:58 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). And likewise for deeper (older) snapshots. sDaniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Olaf Weiser" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA (of course.. only as long nobody changed the file since the snapshot was taken) From: "Alexander Wolf" To: gpfsug-discuss at spectrumscale.org Date: 11/28/2019 07:03 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Gr??en / Kind regards IBM Spectrum Scale Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug main discussion list" Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 14:00 Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob wrote: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question |------------------------------------------------------------------------------| |Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial| | | |------------------------------------------------------------------------------| Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=6F5tNsgC3a8tnTmVCVxGFJgNmWLUYm_MUO-zYDgeQ5o&s=F1aJcNbQxqosWD5WimhGS19MpeszOyIp9CxuUib-c2Q&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14065652.gif Type: image/gif Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14488338.gif Type: image/gif Size: 6645 bytes Desc: not available URL: From zhouhz at cn.ibm.com Fri Nov 29 01:55:31 2019 From: zhouhz at cn.ibm.com (Hai Zhong HZ Zhou) Date: Fri, 29 Nov 2019 01:55:31 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361973.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361974.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361975.png Type: image/png Size: 1134 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Fri Nov 29 07:22:56 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Fri, 29 Nov 2019 07:22:56 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , Message-ID: Hi Thanks very much everyone for this. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Hai Zhong HZ Zhou Sent: 29 November 2019 01:55 To: IBM Spectrum Scale Cc: gpfsug-discuss-bounces at spectrumscale.org ; gpfsug main discussion list ; Leo Luan Subject: Re: [gpfsug-discuss] Compression question Caution - This email from zhouhz at cn.ibm.com originated outside Imperial The statement from Olaf and Alex in below emails are correct. Firstly, compressing and decompressing files in active file system doesn't not trigger data blocks copy-on-write, that is just deallocating the unnecessary original data blocks and put compressed data in few data blocks, and no data blocks copy to snapshot. Secondly, when reading the file from snapshot, it will be redirected to active file system because there's no data blocks in snapshot, and then doing decompression in-memory for snapshot read, while the data on disk is still kept compressed. Regards, Hai Zhong Zhou -----Huzefa H Pancha/India/IBM wrote: ----- To: Hai Zhong HZ Zhou/China/IBM at IBMCN From: IBM Spectrum Scale/Poughkeepsie/IBM Sent by: Huzefa H Pancha/India/IBM Date: 11/29/2019 02:33AM Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list >, Leo Luan/Almaden/IBM at IBMUS Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] Compression question Hai Zhong, Can you please help the customer with their compression related query. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Daniel Kidger" ---28-11-2019 20:58:12---Olaf's explanation makes excellent sense. So is this defi]"Daniel Kidger" ---28-11-2019 20:58:12---Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshot From: "Daniel Kidger" > To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 28-11-2019 20:58 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). And likewise for deeper (older) snapshots. sDaniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Olaf Weiser" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA (of course.. only as long nobody changed the file since the snapshot was taken) From: "Alexander Wolf" > To: gpfsug-discuss at spectrumscale.org Date: 11/28/2019 07:03 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Gr??en / Kind regards [cid:1574992361973] [IBM Spectrum Scale] Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com [cid:1574992361975] IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug main discussion list" > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 14:00 Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob > wrote: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Daniel Kidger > Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Alexander Wolf" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list > Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361973.png Type: image/png Size: 1134 bytes Desc: Image.1574992361973.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361974.png Type: image/png Size: 6645 bytes Desc: Image.1574992361974.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361975.png Type: image/png Size: 1134 bytes Desc: Image.1574992361975.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From kraemerf at de.ibm.com Fri Nov 29 08:35:22 2019 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Fri, 29 Nov 2019 09:35:22 +0100 Subject: [gpfsug-discuss] FYI - Nvidia Magnum IO SC19 Accelerating data for NVIDIA GPUs with IBM Spectrum Scale Message-ID: Nvidia Magnum IO SC19 Accelerating data for NVIDIA GPUs with IBM Spectrum Scale NVIDIA announced the Magnum IO solution that complements leading-edge data management systems such as IBM Spectrum Scale and helps address AI and big data analytics I/O challenges. https://news.developer.nvidia.com/magnum-io-software-suite/ https://youtu.be/_iIAEflTZuE Accelerating data for NVIDIA GPUs - IBM IT Infrastructure Blog https://www.ibm.com/blogs/systems/accelerating-data-for-nvidia-gpus/ -frank- P.S. Oracle Cloud is now using Spectrum Scale :-) https://blogs.oracle.com/cloud-infrastructure/the-fastest-file-servers-in-the-public-cloud P.S. and the winner is ?! CM> "Coldago is a relatively little-known analyst firm and its output stands up well against prominent research firms such as Gartner and Forrester." https://blocksandfiles.com/2019/11/28/coldago-research-26-file-storage-suppliers/ Frank Kraemer IBM Consulting IT Specialist / Client Technical Architect Am Weiher 24, 65451 Kelsterbach, Germany mailto:kraemerf at de.ibm.com Mobile +49171-3043699 IBM Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Nov 1 23:40:31 2019 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 1 Nov 2019 19:40:31 -0400 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= Message-ID: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> How can I force secondary processes to use the folder instructed by the -g option? I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 -P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From spectrumscale at kiranghag.com Sat Nov 2 03:35:32 2019 From: spectrumscale at kiranghag.com (KG) Date: Sat, 2 Nov 2019 09:05:32 +0530 Subject: [gpfsug-discuss] Windows + LDAP + Linux clients interop Message-ID: Hi Folks I need to integrate Windows 10 clients into Spectrum Scale cluster that will have Linux clients. - The Linux clients get authenticated via LDAP - The Windows 10 clients authenticate via AD - There would not be any folder with simultaneous Win/Lin access however the ACLS are required to protect file/directory access amongst users I would like to know how file permissions and overall interop would happen. Do I need to change Win10 client to authenticate via LDAP? OR can they use LDAP while connecting and using GPFS mounts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Nov 4 01:24:35 2019 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sun, 3 Nov 2019 20:24:35 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= In-Reply-To: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> References: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> Message-ID: Please show us the 2 or 3 mmbackup commands that you would like to run concurrently. Peeking into the script, I find: if [[ $scope == "inode-space" ]] then deviceSuffix="${deviceName}.${filesetName}" else deviceSuffix="${deviceName}" I believe mmbackup is designed to allow concurrent backup of different independent filesets within the same filesystem, Or different filesystems... And a single mmbackup instance can drive several TSM servers, which can be named with an option or in the dsm.sys file: # --tsm-servers TSMserver[,TSMserver...] # List of TSM servers to use instead of the servers in the dsm.sys file. From: Jaime Pinto To: gpfsug main discussion list Date: 11/01/2019 07:40 PM Subject: [EXTERNAL] [gpfsug-discuss] mmbackup ?g GlobalWorkDirectory not being followed Sent by: gpfsug-discuss-bounces at spectrumscale.org How can I force secondary processes to use the folder instructed by the -g option? I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 -P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8D_-g0uvUjmBxM3FoCM3Jru71qum_AlcQYOe2gaC9Iw&s=rKNspohw_8ulLhO5Epvqly_vRyiBLxylBWPNKkea2eU&e= ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8D_-g0uvUjmBxM3FoCM3Jru71qum_AlcQYOe2gaC9Iw&s=Yahjw-3p5PGqhgawsVqB2LuwZ151Fov398camDm4XwY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From pinto at scinet.utoronto.ca Mon Nov 4 01:56:35 2019 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sun, 3 Nov 2019 20:56:35 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= In-Reply-To: References: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> Message-ID: <5bf94749-4add-c4f4-63df-21551c5111e1@scinet.utoronto.ca> On 11/3/2019 20:24:35, Marc A Kaplan wrote: > Please show us the 2 or 3 mmbackup commands that you would like to run concurrently. Hey Marc, They would be pretty similar, with the only different being the target TSM server, determined by sourcing a different dsmenv1(2 or 3) prior to the start of each instance, each with its own dsm.sys (3 wrappers). (source dsmenv1; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg1 --scope inodespace -v -a 8 -L 2) (source dsmenv3; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg2 --scope inodespace -v -a 8 -L 2) (source dsmenv3; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg3 --scope inodespace -v -a 8 -L 2) I was playing with the -L (to control the policy), but you bring up a very good point I had not experimented with, such as a single traverse for multiple target servers. It may be just what I need. I'll try this next. Thank you very much, Jaime > > Peeking into the script, I find: > > if [[ $scope == "inode-space" ]] > then > deviceSuffix="${deviceName}.${filesetName}" > else > deviceSuffix="${deviceName}" > > > I believe mmbackup is designed to allow concurrent backup of different independent filesets within the same filesystem, Or different filesystems... > > And a single mmbackup instance can drive several TSM servers, which can be named with an option or in the dsm.sys file: > > # --tsm-servers TSMserver[,TSMserver...] > # List of TSM servers to use instead of the servers in the dsm.sys file. > > > > Inactive hide details for Jaime Pinto ---11/01/2019 07:40:47 PM---How can I force secondary processes to use the folder instrucJaime Pinto ---11/01/2019 07:40:47 PM---How can I force secondary processes to use the folder instructed by the -g option? I started a mmbac > > From: Jaime Pinto > To: gpfsug main discussion list > Date: 11/01/2019 07:40 PM > Subject: [EXTERNAL] [gpfsug-discuss] mmbackup ?g GlobalWorkDirectory not being followed > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > > > How can I force secondary processes to use the folder instructed by the -g option? > > I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) > > However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. > > See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): > > DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 ?-P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace > > > Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. > > Thanks > Jaime > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From heinrich.billich at id.ethz.ch Mon Nov 4 17:13:50 2019 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Mon, 4 Nov 2019 17:13:50 +0000 Subject: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Message-ID: Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Nov 4 19:21:26 2019 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 4 Nov 2019 21:21:26 +0200 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=Mg9Yxn6axX4-iDSDZNIhF58cDheSv41MfR8uVXS_q58&s=bwXBF_0oFwkRREwv9IUvhXGgbQtjhEnJR5Xma7_XFIU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From sadaniel at us.ibm.com Mon Nov 4 19:43:18 2019 From: sadaniel at us.ibm.com (Steven Daniels) Date: Mon, 4 Nov 2019 12:43:18 -0700 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=LfJIsA_kqUNu0c81SWoB1avoQI9WhsKRAILIZcOW8ts&s=0CsyJjJK2f21MXg1WtTCrGNLrg31sqbejXCMQzLyNQo&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From YARD at il.ibm.com Tue Nov 5 11:42:34 2019 From: YARD at il.ibm.com (Yaron Daniel) Date: Tue, 5 Nov 2019 13:42:34 +0200 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: Hi Steven Can you please try to run mmlsnsd - and let me know if you can change the nsd name before/after configure them with mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Steven Daniels" To: gpfsug main discussion list Date: 04/11/2019 21:44 Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=EzTElY1rKKm1TuasJlFc6kJOA5qcwp0FPH71ofd8CsA&s=7Tbbpw_RbqYWzzoaTVvo7V_11FP9ytTQRHw_TWgC24Q&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From scale at us.ibm.com Tue Nov 5 13:51:45 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 5 Nov 2019 08:51:45 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: As far as I know there is no way to change nsd names once they have been created, and yes mmvdisk automatically generates the nsd names. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/05/2019 06:42 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Steven Can you please try to run mmlsnsd - and let me know if you can change the nsd name before/after configure them with mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Steven Daniels" To: gpfsug main discussion list Date: 04/11/2019 21:44 Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=JZEHO0Ia1aoBD_GuPcVjZWgvVTYFsFL5YXqg-0nZCbw&s=iN3GzQg9lDBGVOeJR99XKrc7spI5ZQWGat6gNX5bN8s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Tue Nov 5 18:04:30 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 5 Nov 2019 18:04:30 +0000 Subject: [gpfsug-discuss] Reminder: Sign up for the SSUG meeting at SC19 if you plan to attend Message-ID: Reminder for all those going to SC19 and plan on attending the Spectrum Scale User Group meeting on Sunday November 17th - please pre-register! Agenda details and registration link is here: https://www.spectrumscaleug.org/event/spectrum-scale-user-group-meeting-sc19/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Nov 6 13:06:56 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Wed, 6 Nov 2019 13:06:56 +0000 Subject: [gpfsug-discuss] CIUK Call for Speakers Message-ID: Hi All, It?s just a month until CIUK and as in the past we?re running a Spectrum Scale user group as part of the conference. We?ll be running in the morning of 5th December and are looking for a couple of speakers who could do user talks for the meeting. Please let me know if you are interested in doing a site update/talk on something there! As usual, for this user group you must be a registered CIUK attendee. Thanks Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Fri Nov 8 16:25:24 2019 From: damir.krstic at gmail.com (Damir Krstic) Date: Fri, 8 Nov 2019 10:25:24 -0600 Subject: [gpfsug-discuss] gpfs client and turning off swap Message-ID: I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. Let me know when you get a chance. Thank you. Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_johnson at brown.edu Fri Nov 8 16:31:20 2019 From: david_johnson at brown.edu (david_johnson at brown.edu) Date: Fri, 8 Nov 2019 11:31:20 -0500 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: References: Message-ID: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> We have most of our clients network booted and diskless ? no swap possible. Gpfs still works until someone runs the node out of memory.... -- ddj Dave Johnson > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > ? > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > Let me know when you get a chance. > > Thank you. > Damir > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From damir.krstic at gmail.com Fri Nov 8 16:37:03 2019 From: damir.krstic at gmail.com (Damir Krstic) Date: Fri, 8 Nov 2019 10:37:03 -0600 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: Thanks - that's what I thought. We also (many refreshes ago) ran diskless images with RedHat 6 and GPFS 3.4 and ran into no issues with having swap off. I figured to ask in case something has changed with the Spectrum Scale. Damir On Fri, Nov 8, 2019 at 10:31 AM wrote: > We have most of our clients network booted and diskless ? no swap > possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic > wrote: > > > > ? > > I was wondering if it's safe to turn off swap on gpfs client machines? > we have a case where checkpointing is swapping and we would like to prevent > it from doing so by disabling swap. However, the gpfs manual admin. manual > states to have swap enabled and sufficiently large, but it does not > distinguish between clients and IO servers. > > > > Let me know when you get a chance. > > > > Thank you. > > Damir > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Nov 8 16:44:47 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 8 Nov 2019 16:44:47 +0000 Subject: [gpfsug-discuss] SSUG meeting at SC19 - Important Updates! Message-ID: <6BEA6CAB-A84E-4C1E-8A56-F600F04C4082@nuance.com> For those of you going to (or planning on going) to the User Group meeting at SC19, here is some important updates: You should pre-register here: https://www.spectrumscaleug.org/event/spectrum-scale-user-group-meeting-sc19/ Box Lunches: Thanks to the generous supports of Mark III Systems and Lenovo, we will have box lunches available. These should arrive about 11:30AM on Sunday 11/17. There is a limited number (100) and will include a variety of sandwiches, and some vegetarian selections. Drinks are NOT included. First come, first serve. These are free of charge, and are available to any of those attending either the morning or afternoon session(s). Wi-Fi: Despite the best efforts of the Principals, Sponsors, and IBM, we haven?t been able to make this work. There WILL be a cable drop to the podium for presentations, but no public Wi-Fi access - Hotel Guest Wi-Fi should be available. Looking forward to seeing everyone at SC19! Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Fri Nov 8 16:49:21 2019 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Fri, 8 Nov 2019 16:49:21 +0000 Subject: [gpfsug-discuss] GPFS diskless/stateless clients (was: Re: gpfs client and turning off swap) In-Reply-To: References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: <229B3495-6D3D-40DF-8927-0960F751749C@rutgers.edu> We run stateless machines today with CentOS 7.x and have run GPFS 4.1.0.8, 4.2.3.x, and 5.0.x. We don?t use swap either in most cases, and have never needed to do anything special for that. We?ve generally needed to rig ExecStartPre stuff in systemd in order to make sure that the machines could restore their cluster membership. If anyone else can share how they do that, it might be interesting information. In our case, this works (our provisioning system writes these variables as the actual host names) ? we used to put this in an override to gpfs.service; I believe mmautoload.service is relatively new (either in 4.2.3.x to 5.0.x or 4.1.x to 4.2.x). [root at test mmautoload.service.d]# pwd /etc/systemd/system/mmautoload.service.d [root at test mmautoload.service.d]# cat override.conf [Unit] Before=slurmd.service After=sys-subsystem-net-devices-ib0.device [Service] ExecStartPre=/usr/lpp/mmfs/bin/mmsdrrestore -p $CLUSTERMGR -R /usr/bin/scp ExecStartPre=/usr/lpp/mmfs/bin/mmauth genkey propagate -N $NODENAME -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' > On Nov 8, 2019, at 11:37 AM, Damir Krstic wrote: > > Thanks - that's what I thought. We also (many refreshes ago) ran diskless images with RedHat 6 and GPFS 3.4 and ran into no issues with having swap off. I figured to ask in case something has changed with the Spectrum Scale. > Damir > > On Fri, Nov 8, 2019 at 10:31 AM wrote: > We have most of our clients network booted and diskless ? no swap possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > > > ? > > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > > > Let me know when you get a chance. > > > > Thank you. > > Damir > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From skylar2 at uw.edu Fri Nov 8 16:52:33 2019 From: skylar2 at uw.edu (Skylar Thompson) Date: Fri, 8 Nov 2019 16:52:33 +0000 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> We don't run diskless systems, but use cgroups where possible to limit the damage users can do to a node. We derate the node's total usable memory by 2GB (OS) + GPFS pagepool to avoid paging whenever possible. On Fri, Nov 08, 2019 at 11:31:20AM -0500, david_johnson at brown.edu wrote: > We have most of our clients network booted and diskless ??? no swap possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > > > ??? > > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > > > Let me know when you get a chance. -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From valdis.kletnieks at vt.edu Fri Nov 8 21:15:12 2019 From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks) Date: Fri, 08 Nov 2019 16:15:12 -0500 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: References: Message-ID: <30980.1573247712@turing-police> On Fri, 08 Nov 2019 10:25:24 -0600, Damir Krstic said: > I was wondering if it's safe to turn off swap on gpfs client machines? we > have a case where checkpointing is swapping and we would like to prevent it > from doing so by disabling swap. However, the gpfs manual admin. GPFS will work just fine without a swap space if the system has sufficient RAM. However, if checkpointing is swapping, you don't have enough RAM (at least with your current configuration), and turning off swap will result in processes being killed due to lack of memory. You *might* get it to work by tuning system config variables to reduce RAM consumption. But that's better tested while you still have swap defined, and not removing swap until you have a system not needing it. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Sat Nov 9 09:27:10 2019 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 9 Nov 2019 09:27:10 +0000 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> Message-ID: On 08/11/2019 16:52, Skylar Thompson wrote: > We don't run diskless systems, but use cgroups where possible to limit the > damage users can do to a node. We derate the node's total usable memory by > 2GB (OS) + GPFS pagepool to avoid paging whenever possible. > I would add that as the RAM in the node rises swap becomes pointless. We find that it's a bit pointless at 192GB of RAM in a node, for our large memory nodes which have 3TB of RAM, swap is utterly pointless. In fact there is more RAM than there is disk space. In either case the old rules about setting swap to be twice RAM or even swap equals RAM are either hard to achieve to impossible. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From julian.jakobs at cec.mpg.de Mon Nov 11 11:25:29 2019 From: julian.jakobs at cec.mpg.de (Jakobs, Julian) Date: Mon, 11 Nov 2019 11:25:29 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 Message-ID: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Hello, has anyone already tried Spectrum Scale on RHEL 8.1? I can see from the GPFS FAQ that "RHEL 8" (no minor version indicated) is supported as of the 5.0.4 release. However latest kernel level fully tested is 4.18.0-80, indicating that only RHEL 8.0 was tested. I tested an installation with 8.1 (kernel version 4.18.0-147) and mmbuildgpl failed due to errors while compiling the gpl (incompatible pointer type). Is this expected behaviour or is there maybe something else wrong with the installation? If this needs a new GPFS release, is there an estimated time? I would much prefer to install it with RHEL 8.1 due to 8.0 not being a EUS release. Best, Julian Jakobs -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6220 bytes Desc: not available URL: From stockf at us.ibm.com Mon Nov 11 12:47:28 2019 From: stockf at us.ibm.com (Frederick Stock) Date: Mon, 11 Nov 2019 12:47:28 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> References: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Nov 11 13:25:27 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 11 Nov 2019 13:25:27 +0000 Subject: [gpfsug-discuss] Force removal of problematic inode? Message-ID: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> Any ideas on how I could get out of this, short of a full mmfsck? Directory paths shortened to obfuscate data. [root at nrg5-gpfs01 remove]# tsfindinode -i 11198927 /gpfs/lm gpfs_iopen(/gpfs/lm/.../build_respell_map)(184769551:0xD4F8): Device or resource busy ^C ls -li /gpfs/lm/?/ ls: cannot access /gpfs/lm/.../build_respell_map: Device or resource busy total 0 ? d????????? ? ? ? ? ? build_respell_map [root at nrg5-gpfs01 remove]# rm -rf /gpfs/lm/.../* rm: cannot remove ?/gpfs/lm/.../build_respell_map?: Device or resource busy Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Mon Nov 11 13:28:19 2019 From: stockf at us.ibm.com (Frederick Stock) Date: Mon, 11 Nov 2019 13:28:19 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: References: , <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: From A.Wolf-Reber at de.ibm.com Mon Nov 11 13:30:09 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Mon, 11 Nov 2019 13:30:09 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: References: , <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746414.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746415.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746416.png Type: image/png Size: 1134 bytes Desc: not available URL: From alexander.zimmermann at id.ethz.ch Mon Nov 11 14:16:34 2019 From: alexander.zimmermann at id.ethz.ch (Zimmermann Alexander (ID SD)) Date: Mon, 11 Nov 2019 14:16:34 +0000 Subject: [gpfsug-discuss] orphaned deleted fileset Message-ID: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> Hi, we've an orphaned deleted fileset in one of our filesystems: [azi4ea at nas22ems01 ~]$ mmlsfileset /dev/fs2201 --deleted Filesets in file system 'fs2201': Name Status Path (chab_services_admin_s1) Deleted /fs2201/.snapshots/@GMT-2019.10.21-10.45.55/chab_services_admin_s1 (biol_bc_azzalin_1) Deleted -- [azi4ea at nas22ems01 ~]$ biol_bc_azzalin_1 is now in that state for roughly 1 week, initially it was in the state Deleting until I ran (another?) mmdelfileset command on it. The order to delete it in our automation tool was completed in July so it's hard find out what went wrong back then. Will it get cleaned up on its own and I'm only too impatient or do I need to intervene to get rid of it completely? Many thanks in advance for your aid! Kind regards, Alexander Zimmermann Alexander Zimmermann ETH Z?rich ID Systemdienste - Speicher Weinbergstrasse 11, WEC C16 8092 Z?rich Tel.: +41 44 632 72 39 E-Mail: alexander.zimmermann at id.ethz.ch www.id.ethz.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From will.schmied at stjude.org Mon Nov 11 22:02:19 2019 From: will.schmied at stjude.org (Schmied, Will) Date: Mon, 11 Nov 2019 22:02:19 +0000 Subject: [gpfsug-discuss] Job opportunity: HPC Storage Architect at St. Jude Message-ID: <1E710D36-37F5-4BCC-BE23-8A3A192B67BA@stjude.org> Hello everyone, St. Jude Children?s Research Hospital (Memphis, TN) has recently posted a job opening for a HPC Storage Architect, a senior level position working primarily to operate and maintain multiple Spectrum Scale clusters in support of research and other HPC workloads. You can view the job posting, and begin your application, here: http://myjob.io/nd6qd You can find all jobs, and information about working at St. Jude, here: https://www.stjude.org/jobs/hospital.html Please feel free to contact me directly off list if you have any questions. I?ll be at SC this year, and also presenting at the user group meeting, and hope to see you there. Thanks, Will ________________________________ Email Disclaimer: www.stjude.org/emaildisclaimer Consultation Disclaimer: www.stjude.org/consultationdisclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Tue Nov 12 08:16:29 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 12 Nov 2019 16:16:29 +0800 Subject: [gpfsug-discuss] Force removal of problematic inode? In-Reply-To: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> References: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> Message-ID: You can try if restarting GPFS daemon would help. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 11/11/2019 09:28 PM Subject: [EXTERNAL] [gpfsug-discuss] Force removal of problematic inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org Any ideas on how I could get out of this, short of a full mmfsck? Directory paths shortened to obfuscate data. [root at nrg5-gpfs01 remove]# tsfindinode -i 11198927 /gpfs/lm gpfs_iopen(/gpfs/lm/.../build_respell_map)(184769551:0xD4F8): Device or resource busy ^C ls -li /gpfs/lm/?/ ls: cannot access /gpfs/lm/.../build_respell_map: Device or resource busy total 0 ? d????????? ? ? ? ? ? build_respell_map [root at nrg5-gpfs01 remove]# rm -rf /gpfs/lm/.../* rm: cannot remove ?/gpfs/lm/.../build_respell_map?: Device or resource busy Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=Pid3Qt9isGIuPoObkvy-RLwrhUmEM-mNU4YAfNWpDRo&s=TMRis9DxrIhg7D9xwf4IYi0tNH1RrWqmeSL1cDD1hi4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From scale at us.ibm.com Tue Nov 12 08:18:24 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 12 Nov 2019 16:18:24 +0800 Subject: [gpfsug-discuss] orphaned deleted fileset In-Reply-To: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> References: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> Message-ID: run mmfsck or try to delete another fileset to see if that could trigger cleanup. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Zimmermann Alexander (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Date: 11/11/2019 10:16 PM Subject: [EXTERNAL] [gpfsug-discuss] orphaned deleted fileset Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, we?ve an orphaned deleted fileset in one of our filesystems: [azi4ea at nas22ems01 ~]$ mmlsfileset /dev/fs2201 --deleted Filesets in file system 'fs2201': Name Status Path (chab_services_admin_s1) Deleted /fs2201/.snapshots/@GMT-2019.10.21-10.45.55/chab_services_admin_s1 (biol_bc_azzalin_1) Deleted -- [azi4ea at nas22ems01 ~]$ biol_bc_azzalin_1 is now in that state for roughly 1 week, initially it was in the state Deleting until I ran (another?) mmdelfileset command on it. The order to delete it in our automation tool was completed in July so it?s hard find out what went wrong back then. Will it get cleaned up on its own and I?m only too impatient or do I need to intervene to get rid of it completely? Many thanks in advance for your aid! Kind regards, Alexander Zimmermann Alexander Zimmermann ETH Z?rich ID Systemdienste - Speicher Weinbergstrasse 11, WEC C16 8092 Z?rich Tel.: +41 44 632 72 39 E-Mail: alexander.zimmermann at id.ethz.ch www.id.ethz.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=dy9kUmE1i9dh2VjQBu8Cqis2Us0lfMxVc_OzA2Jwhfk&s=eazf8OnQPmH23k5Z0VwYOGDLa5-LL88CcvdUqctGMIY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From S.J.Thompson at bham.ac.uk Tue Nov 12 14:49:07 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Tue, 12 Nov 2019 14:49:07 +0000 Subject: [gpfsug-discuss] SC19 SSUG Lightning talks Message-ID: <5C96F4C4-13B8-41A7-8834-9D32928AE738@bham.ac.uk> Hi All, It?s less than a week until the Spectrum Scale user group at SC19 in Denver! As part of the agenda, we have included a lightning talk slot and I?m looking for people to volunteer to do a zero slide talk on something related to Scale. Those at the UK event may remember we ran something similar on ?my favourite/worst feature of Spectrum Scale?. I?m happy to use the same topic, or for people to do a 3-5 minute lightning slot on Scale and why they use it ? or some other Scale topic entirely! If you are interested, I?ll be around on Sunday at the SSUG and you can let me know then/volunteer in the slot. If no-one volunteers, we?ll have a very quiet 30 mins in the user group session! Please have a think on something you might want to say on the day! Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From Rafael.Cezario at ibm.com Wed Nov 13 17:24:34 2019 From: Rafael.Cezario at ibm.com (Rafael Cezario) Date: Wed, 13 Nov 2019 14:24:34 -0300 Subject: [gpfsug-discuss] Online migration Message-ID: Hello, I have a Spectrum Scale cluster with 10 nodes on the version 4.2. The cluster have maxblocksize=DEFAULT. I will migrate all nodes to 5.0.3 (online) none downtime. But the maxblocksize is fixed on the mmlsconfig after the migration (the man have this information) maxblocksize 1M man mmchconfig Note: When you migrate a cluster from an earlier version to 5.0.0 or later, the value of maxblocksize stays the same. However, if maxblocksize was set to DEFAULT in the earlier version of the cluster, then migrating it to 5.0.0 or later sets it explicitly to 1 MiB, which was the default value in earlier versions. To change maxblocksize to the default value after migrating to 5.0.0 or later, set maxblocksize=DEFAULT (4 MiB). How Can I perform non-stop migration and after migration create a new filesystem with blocksize = 4MiB? My question is about finishing the migration and create a new filesystem with block size set to 4 MiB, its won't be possible until the maxblocksize changes. In a environment where all migration will be done without stopping the cluster, my question was if there was any alternative This behavior forces the entire cluster to stop. Regards, Rafael Cez?rio Dam?sio Cunha IT Specialist - PowerVM/AIX/Spectrum Scale Technology Support Services Global Technology Services Mobile: 55-6198122-4435 E-mail: rafael.cezario at ibm.com Scn Quadra 4, Bloco B, n.100 Brasilia, DF 70714-900 Brazil -------------- next part -------------- An HTML attachment was scrubbed... URL: From knop at us.ibm.com Wed Nov 13 19:05:00 2019 From: knop at us.ibm.com (Felipe Knop) Date: Wed, 13 Nov 2019 19:05:00 +0000 Subject: [gpfsug-discuss] Online migration In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From andi at christiansen.xxx Wed Nov 13 21:33:57 2019 From: andi at christiansen.xxx (Andi Christiansen) Date: Wed, 13 Nov 2019 22:33:57 +0100 (CET) Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. Message-ID: <384482306.75861.1573680837216@privateemail.com> An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Nov 13 22:05:08 2019 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 13 Nov 2019 22:05:08 +0000 Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. In-Reply-To: <384482306.75861.1573680837216@privateemail.com> References: <384482306.75861.1573680837216@privateemail.com> Message-ID: Hello, This is simply due to the fact that your ?test? is doing parallel appended writes to the same file. With GPFS, there is one node at a time that can write to roughly the same position (aka byte range) in a file. This is handled by something called a write_lock token that has to be revoked on the node that currently owns the token, then given to one of the other nodes attempting to perform the write. This overhead in lock token management is done for almost every write that is happening in this ?test?. This is not something that you would ever really want an application to do anyways. GPFS offers byte-range locking to allow for very fast, non-blocking I/O to the same file from multiple nodes in parallel. So if you change your ?test? so that each node seeks to a different part of the single file and that each node writes into the file in a way that there is no overlap in the byte range of your total writes from the node, then GPFS will give a byte-range write_lock token to each node. Since no node will attempt to write to the same part of the file as another node (which is what you?d want anyways in most cases) then there will be no write_lock token revoking and no blocking of your I/O to the single file. Hope that helps, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Andi Christiansen Sent: Wednesday, November 13, 2019 3:34 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. [EXTERNAL EMAIL] Hi all, I am in the process of replacing a beegfs cluster with a spectrum scale cluster and some of our initial tests have returned poor performance when writing from multiple client nodes to the same file. If we setup a client to write into a file it takes less than 8 seconds to complete the write on Flash and about the same on NLSAS storage. But if we get 3 more nodes to do the exact same write the cluster seems to slow down and completes all writes in about 1.5 minute. We are running 5.0.4-0 on 4 Lenovo SR630 servers with a V7000 control enclosure with flash drives and a 92F drawer with NLSAS drives behind the nodes attach through SAN. Is there something I am missing since the writes are so slow compared to beegfs? Beegfs completes the write from all clients within 9 second when run from 4 clients doing the same write to the same file GPFS takes 1.5 min to do the same. Tests run: time (for i in `seq 5000`;do echo "This is $(hostname) writing line number" $i >> "/gpfs_T0/test/benchmark1.txt";done) ########## this is run from 4 gpfs client nodes at the same time. Result for scale: real 1m43.995s user 0m0.821s sys 0m3.545s Result for beegfs: real 0m6.507s user 0m0.651s sys 0m2.534s if we run writes from each client node to separate files, performance is way better with GPFS than beegfs but not when the writes are done parallel. If anyone have an idea I would be glad to hear it ? Best Regards Andi Christiansen -------------- next part -------------- An HTML attachment was scrubbed... URL: From tshukla at NYGENOME.ORG Wed Nov 20 15:36:16 2019 From: tshukla at NYGENOME.ORG (Tushit Shukla) Date: Wed, 20 Nov 2019 15:36:16 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 81, Issue 43 In-Reply-To: References: Message-ID: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> Lohit, Did you get any progress on file system performance issue with large block size ? Apparently we are also going through the same issue after we migrated to 16M file system (from 4MB filesystem). Only thing which gives some relief is when we increase pagepool to 32G. We also tried playing with prefetchAggressivenessRead parameter by reducing it to 1 but not sure if it is doing anything. thanks ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 22, 2018 12:18:58 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 81, Issue 43 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: GPFS, Pagepool and Block size -> Perfomance reduces with larger block size (Sven Oehme) ---------------------------------------------------------------------- Message: 1 Date: Mon, 22 Oct 2018 09:18:43 -0700 From: Sven Oehme To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS, Pagepool and Block size -> Perfomance reduces with larger block size Message-ID: Content-Type: text/plain; charset="utf-8" oops, somehow that slipped my inbox, i only saw that reply right now. its really hard to see from a trace snipped if the lock is the issue as the lower level locks don't show up in default traces. without having access to source code and a detailed trace you won't make much progress here. sven On Thu, Sep 27, 2018 at 12:31 PM wrote: > Thank you Sven, > > Turning of prefetching did not improve the performance, but it did degrade > a bit. > > I have made the prefetching default and took trace dump, for tracectl with > trace=io. Let me know if you want me to paste/attach it here. > > May i know, how could i confirm if the below is true? > > 1. this could be serialization around buffer locks. as larger your >>> blocksize gets as larger is the amount of data one of this pagepool buffers >>> will maintain, if there is a lot of concurrency on smaller amount of data >>> more threads potentially compete for the same buffer lock to copy stuff in >>> and out of a particular buffer, hence things go slower compared to the same >>> amount of data spread across more buffers, each of smaller size. >>> >>> > Will the above trace help in understanding if it is a serialization issue? > > I had been discussing the same with GPFS support for past few months, and > it seems to be that most of the time is being spent at cxiUXfer. They could > not understand on why it is taking spending so much of time in cxiUXfer. I > was seeing the same from perf top, and pagefaults. > > Below is snippet from what the support had said : > > ???????????????????????????? > > I searched all of the gpfsRead from trace and sort them by spending-time. > Except 2 reads which need fetch data from nsd server, the slowest read is > in the thread 72170. It took 112470.362 us. > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum: 72165 6.860911319 > rdwr 141857.076 us + NSDIO > > trcrpt.2018-08-06_12.26.28.39794.lt15.trsum: 72170 1.483947593 > rdwr 112470.362 us + cxiUXfer > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum: 72165 6.949042593 > rdwr 88126.278 us + NSDIO > > trcrpt.2018-08-06_12.27.03.47706.lt15.trsum: 72156 2.919334474 > rdwr 81057.657 us + cxiUXfer > > trcrpt.2018-08-06_12.23.30.72745.lt15.trsum: 72154 1.167484466 > rdwr 76033.488 us + cxiUXfer > > trcrpt.2018-08-06_12.24.06.7508.lt15.trsum: 72187 0.685237501 > rdwr 70772.326 us + cxiUXfer > > trcrpt.2018-08-06_12.25.17.23989.lt15.trsum: 72193 4.757996530 > rdwr 70447.838 us + cxiUXfer > > > I check each of the slow IO as above, and find they all spend much time in > the function cxiUXfer. This function is used to copy data from kernel > buffer to user buffer. I am not sure why it took so much time. This should > be related to the pagefaults and pgfree you observed. Below is the trace > data for thread 72170. > > > 1.371477231 72170 TRACE_VNODE: gpfs_f_rdwr enter: fP > 0xFFFF882541649400 f_flags 0x8000 flags 0x8001 op 0 iovec > 0xFFFF881F2AFB3E70 count 1 offset 0x168F30D dentry 0xFFFF887C0CC298C0 > private 0xFFFF883F607175C0 iP 0xFFFF8823AA3CBFC0 name '410513.svs' > > .... > > 1.371483547 72170 TRACE_KSVFS: cachedReadFast exit: > uio_resid 16777216 code 1 err 11 > > .... > > 1.371498780 72170 TRACE_KSVFS: kSFSReadFast: oiP > 0xFFFFC90060B46740 offset 0x168F30D dataBufP FFFFC9003645A5A8 nDesc 64 buf > 200043C0000 valid words 64 dirty words 0 blkOff 0 > > 1.371499035 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate begin ul 0xFFFFC900333F1A40 holdCount 0 > ioType 0x2 inProg 0x15 > > 1.371500157 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate ul 0xFFFFC900333F1A40 holdCount 0 ioType 0x2 > inProg 0x16 err 0 > > 1.371500606 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 toIOBuf 0 offset 6877965 len > 9899251 > > 1.371500793 72170 TRACE_KSVFS: cxiUXfer: ndesc 0 skip > dataAddrP 0x200043C0000 currOffset 0 currLen 262144 bufOffset 6877965 > > .... > > 1.371505949 72170 TRACE_KSVFS: cxiUXfer: ndesc 25 skip > dataAddrP 0x2001AF80000 currOffset 6553600 currLen 262144 bufOffset 6877965 > > 1.371506236 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > currOffset 6815744 tmpLen 262144 dataAddrP 0x2001AFCF30D currLen 199923 > pageOffset 781 pageLen 3315 plP 0xFFFF887F7B90D600 > > 1.373649823 72170 TRACE_KSVFS: cxiUXfer: nDesc 27 > currOffset 7077888 tmpLen 262144 dataAddrP 0x20027400000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.375158799 72170 TRACE_KSVFS: cxiUXfer: nDesc 28 > currOffset 7340032 tmpLen 262144 dataAddrP 0x20027440000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.376661566 72170 TRACE_KSVFS: cxiUXfer: nDesc 29 > currOffset 7602176 tmpLen 262144 dataAddrP 0x2002C180000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.377892653 72170 TRACE_KSVFS: cxiUXfer: nDesc 30 > currOffset 7864320 tmpLen 262144 dataAddrP 0x2002C1C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > .... > > 1.471389843 72170 TRACE_KSVFS: cxiUXfer: nDesc 62 > currOffset 16252928 tmpLen 262144 dataAddrP 0x2001D2C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.471845629 72170 TRACE_KSVFS: cxiUXfer: nDesc 63 > currOffset 16515072 tmpLen 262144 dataAddrP 0x2003EC80000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.472417149 72170 TRACE_KSVFS: cxiDetachIOBuffer: > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 > > 1.472417775 72170 TRACE_LOCK: unlock_vfs: type Data, > key 0000000000000004:000000001B1F24BF:0000000000000001 lock_mode have ro > token xw lock_state old [ ro:27 ] new [ ro:26 ] holdCount now 27 > > 1.472418427 72170 TRACE_LOCK: hash tab lookup vfs: > found cP 0xFFFFC9005FC0CDE0 holdCount now 14 > > 1.472418592 72170 TRACE_LOCK: lock_vfs: type Data key > 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode want ro status > valid token xw/xw lock_state [ ro:12 ] flags 0x0 holdCount 14 > > 1.472419842 72170 TRACE_KSVFS: kSFSReadFast: oiP > 0xFFFFC90060B46740 offset 0x2000000 dataBufP FFFFC9003643C908 nDesc 64 buf > 38033480000 valid words 64 dirty words 0 blkOff 0 > > 1.472420029 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate begin ul 0xFFFFC9005FC0CF98 holdCount 0 > ioType 0x2 inProg 0xC > > 1.472420187 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate ul 0xFFFFC9005FC0CF98 holdCount 0 ioType 0x2 > inProg 0xD err 0 > > 1.472420652 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 toIOBuf 0 offset 0 len 6877965 > > 1.472420936 72170 TRACE_KSVFS: cxiUXfer: nDesc 0 > currOffset 0 tmpLen 262144 dataAddrP 0x38033480000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.472824790 72170 TRACE_KSVFS: cxiUXfer: nDesc 1 > currOffset 262144 tmpLen 262144 dataAddrP 0x380334C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.473243905 72170 TRACE_KSVFS: cxiUXfer: nDesc 2 > currOffset 524288 tmpLen 262144 dataAddrP 0x38024280000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > .... > > 1.482949347 72170 TRACE_KSVFS: cxiUXfer: nDesc 24 > currOffset 6291456 tmpLen 262144 dataAddrP 0x38025E80000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483354265 72170 TRACE_KSVFS: cxiUXfer: nDesc 25 > currOffset 6553600 tmpLen 262144 dataAddrP 0x38025EC0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483766631 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > currOffset 6815744 tmpLen 262144 dataAddrP 0x38003B00000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483943894 72170 TRACE_KSVFS: cxiDetachIOBuffer: > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 > > 1.483944339 72170 TRACE_LOCK: unlock_vfs: type Data, > key 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode have ro > token xw lock_state old [ ro:14 ] new [ ro:13 ] holdCount now 14 > > 1.483944683 72170 TRACE_BRL: brUnlockM: ofP > 0xFFFFC90069346B68 inode 455025855 snap 0 handle 0xFFFFC9003637D020 range > 0x168F30D-0x268F30C mode ro > > 1.483944985 72170 TRACE_KSVFS: kSFSReadFast exit: > uio_resid 0 err 0 > > 1.483945264 72170 TRACE_LOCK: unlock_vfs_m: type > Inode, key 305F105B9701E60A:000000001B1F24BF:0000000000000000 lock_mode > have ro status valid token rs lock_state old [ ro:25 ] new [ ro:24 ] > > 1.483945423 72170 TRACE_LOCK: unlock_vfs_m: cP > 0xFFFFC90069346B68 holdCount 25 > > 1.483945624 72170 TRACE_VNODE: gpfsRead exit: fast err > 0 > > 1.483946831 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > 0xFFFFC90035E52F78 NotQuiesced vfsOp 2 > > 1.483946975 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > 0xFFFFC90035E52F78 vfsOp 2 users 1-1 > > 1.483947116 72170 TRACE_KSVFS: ReleaseDaemonSegAndSG: > sli 38 count 2 needCleanup 0 > > 1.483947593 72170 TRACE_VNODE: gpfs_f_rdwr exit: fP > 0xFFFF882541649400 total_len 16777216 uio_resid 0 offset 0x268F30D rc 0 > > > ??????????????????????????????????????????? > > > > Regards, > Lohit > > On Sep 19, 2018, 3:11 PM -0400, Sven Oehme , wrote: > > the document primarily explains all performance specific knobs. general > advice would be to longer set anything beside workerthreads, pagepool and > filecache on 5.X systems as most other settings are no longer relevant > (thats a client side statement) . thats is true until you hit strange > workloads , which is why all the knobs are still there :-) > > sven > > > On Wed, Sep 19, 2018 at 11:17 AM wrote: > >> Thanks Sven. >> I will disable it completely and see how it behaves. >> >> Is this the presentation? >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2014_UG10-5FGPFS-5FPerformance-5FSession-5Fv10.pdf&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=-Y1ncRDmgOCfWDQaXj-OcbiM5gcs0LcllAShfNapHtI&e= >> >> I guess i read it, but it did not strike me at this situation. I will try >> to read it again and see if i could make use of it. >> >> Regards, >> Lohit >> >> On Sep 19, 2018, 2:12 PM -0400, Sven Oehme , wrote: >> >> seem like you never read my performance presentation from a few years ago >> ;-) >> >> you can control this on a per node basis , either for all i/o : >> >> prefetchAggressiveness = X >> >> or individual for reads or writes : >> >> prefetchAggressivenessRead = X >> prefetchAggressivenessWrite = X >> >> for a start i would turn it off completely via : >> >> mmchconfig prefetchAggressiveness=0 -I -N nodename >> >> that will turn it off only for that node and only until you restart the >> node. >> then see what happens >> >> sven >> >> >> On Wed, Sep 19, 2018 at 11:07 AM wrote: >> >>> Thank you Sven. >>> >>> I mostly think it could be 1. or some other issue. >>> I don?t think it could be 2. , because i can replicate this issue no >>> matter what is the size of the dataset. It happens for few files that could >>> easily fit in the page pool too. >>> >>> I do see a lot more page faults for 16M compared to 1M, so it could be >>> related to many threads trying to compete for the same buffer space. >>> >>> I will try to take the trace with trace=io option and see if can find >>> something. >>> >>> How do i turn of prefetching? Can i turn it off for a single >>> node/client? >>> >>> Regards, >>> Lohit >>> >>> On Sep 18, 2018, 5:23 PM -0400, Sven Oehme , wrote: >>> >>> Hi, >>> >>> taking a trace would tell for sure, but i suspect what you might be >>> hitting one or even multiple issues which have similar negative performance >>> impacts but different root causes. >>> >>> 1. this could be serialization around buffer locks. as larger your >>> blocksize gets as larger is the amount of data one of this pagepool buffers >>> will maintain, if there is a lot of concurrency on smaller amount of data >>> more threads potentially compete for the same buffer lock to copy stuff in >>> and out of a particular buffer, hence things go slower compared to the same >>> amount of data spread across more buffers, each of smaller size. >>> >>> 2. your data set is small'ish, lets say a couple of time bigger than the >>> pagepool and you random access it with multiple threads. what will happen >>> is that because it doesn't fit into the cache it will be read from the >>> backend. if multiple threads hit the same 16 mb block at once with multiple >>> 4k random reads, it will read the whole 16mb block because it thinks it >>> will benefit from it later on out of cache, but because it fully random the >>> same happens with the next block and the next and so on and before you get >>> back to this block it was pushed out of the cache because of lack of enough >>> pagepool. >>> >>> i could think of multiple other scenarios , which is why its so hard to >>> accurately benchmark an application because you will design a benchmark to >>> test an application, but it actually almost always behaves different then >>> you think it does :-) >>> >>> so best is to run the real application and see under which configuration >>> it works best. >>> >>> you could also take a trace with trace=io and then look at >>> >>> TRACE_VNOP: READ: >>> TRACE_VNOP: WRITE: >>> >>> and compare them to >>> >>> TRACE_IO: QIO: read >>> TRACE_IO: QIO: write >>> >>> and see if the numbers summed up for both are somewhat equal. if >>> TRACE_VNOP is significant smaller than TRACE_IO you most likely do more i/o >>> than you should and turning prefetching off might actually make things >>> faster . >>> >>> keep in mind i am no longer working for IBM so all i say might be >>> obsolete by now, i no longer have access to the one and only truth aka the >>> source code ... but if i am wrong i am sure somebody will point this out >>> soon ;-) >>> >>> sven >>> >>> >>> >>> >>> On Tue, Sep 18, 2018 at 10:31 AM wrote: >>> >>>> Hello All, >>>> >>>> This is a continuation to the previous discussion that i had with Sven. >>>> However against what i had mentioned previously - i realize that this >>>> is ?not? related to mmap, and i see it when doing random freads. >>>> >>>> I see that block-size of the filesystem matters when reading from Page >>>> pool. >>>> I see a major difference in performance when compared 1M to 16M, when >>>> doing lot of random small freads with all of the data in pagepool. >>>> >>>> Performance for 1M is a magnitude ?more? than the performance that i >>>> see for 16M. >>>> >>>> The GPFS that we have currently is : >>>> Version : 5.0.1-0.5 >>>> Filesystem version: 19.01 (5.0.1.0) >>>> Block-size : 16M >>>> >>>> I had made the filesystem block-size to be 16M, thinking that i would >>>> get the most performance for both random/sequential reads from 16M than the >>>> smaller block-sizes. >>>> With GPFS 5.0, i made use the 1024 sub-blocks instead of 32 and thus >>>> not loose lot of storage space even with 16M. >>>> I had run few benchmarks and i did see that 16M was performing better >>>> ?when hitting storage/disks? with respect to bandwidth for >>>> random/sequential on small/large reads. >>>> >>>> However, with this particular workload - where it freads a chunk of >>>> data randomly from hundreds of files -> I see that the number of >>>> page-faults increase with block-size and actually reduce the performance. >>>> 1M performs a lot better than 16M, and may be i will get better >>>> performance with less than 1M. >>>> It gives the best performance when reading from local disk, with 4K >>>> block size filesystem. >>>> >>>> What i mean by performance when it comes to this workload - is not the >>>> bandwidth but the amount of time that it takes to do each iteration/read >>>> batch of data. >>>> >>>> I figure what is happening is: >>>> fread is trying to read a full block size of 16M - which is good in a >>>> way, when it hits the hard disk. >>>> But the application could be using just a small part of that 16M. Thus >>>> when randomly reading(freads) lot of data of 16M chunk size - it is page >>>> faulting a lot more and causing the performance to drop . >>>> I could try to make the application do read instead of freads, but i >>>> fear that could be bad too since it might be hitting the disk with a very >>>> small block size and that is not good. >>>> >>>> With the way i see things now - >>>> I believe it could be best if the application does random reads of >>>> 4k/1M from pagepool but some how does 16M from rotating disks. >>>> >>>> I don?t see any way of doing the above other than following a different >>>> approach where i create a filesystem with a smaller block size ( 1M or less >>>> than 1M ), on SSDs as a tier. >>>> >>>> May i please ask for advise, if what i am understanding/seeing is right >>>> and the best solution possible for the above scenario. >>>> >>>> Regards, >>>> Lohit >>>> >>>> On Apr 11, 2018, 10:36 AM -0400, Lohit Valleru , >>>> wrote: >>>> >>>> Hey Sven, >>>> >>>> This is regarding mmap issues and GPFS. >>>> We had discussed previously of experimenting with GPFS 5. >>>> >>>> I now have upgraded all of compute nodes and NSD nodes to GPFS 5.0.0.2 >>>> >>>> I am yet to experiment with mmap performance, but before that - I am >>>> seeing weird hangs with GPFS 5 and I think it could be related to mmap. >>>> >>>> Have you seen GPFS ever hang on this syscall? >>>> [Tue Apr 10 04:20:13 2018] [] >>>> _ZN10gpfsNode_t8mmapLockEiiPKj+0xb5/0x140 [mmfs26] >>>> >>>> I see the above ,when kernel hangs and throws out a series of trace >>>> calls. >>>> >>>> I somehow think the above trace is related to processes hanging on GPFS >>>> forever. There are no errors in GPFS however. >>>> >>>> Also, I think the above happens only when the mmap threads go above a >>>> particular number. >>>> >>>> We had faced a similar issue in 4.2.3 and it was resolved in a patch to >>>> 4.2.3.2 . At that time , the issue happened when mmap threads go more than >>>> worker1threads. According to the ticket - it was a mmap race condition that >>>> GPFS was not handling well. >>>> >>>> I am not sure if this issue is a repeat and I am yet to isolate the >>>> incident and test with increasing number of mmap threads. >>>> >>>> I am not 100 percent sure if this is related to mmap yet but just >>>> wanted to ask you if you have seen anything like above. >>>> >>>> Thanks, >>>> >>>> Lohit >>>> >>>> On Feb 22, 2018, 3:59 PM -0500, Sven Oehme , wrote: >>>> >>>> Hi Lohit, >>>> >>>> i am working with ray on a mmap performance improvement right now, >>>> which most likely has the same root cause as yours , see --> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_2018-2DJanuary_004411.html&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=AUEL847F_a-j6Y7t1fDMj4j33vLqvI6XrrNCVS5pUyA&e= >>>> the thread above is silent after a couple of back and rorth, but ray >>>> and i have active communication in the background and will repost as soon >>>> as there is something new to share. >>>> i am happy to look at this issue after we finish with ray's workload if >>>> there is something missing, but first let's finish his, get you try the >>>> same fix and see if there is something missing. >>>> >>>> btw. if people would share their use of MMAP , what applications they >>>> use (home grown, just use lmdb which uses mmap under the cover, etc) please >>>> let me know so i get a better picture on how wide the usage is with GPFS. i >>>> know a lot of the ML/DL workloads are using it, but i would like to know >>>> what else is out there i might not think about. feel free to drop me a >>>> personal note, i might not reply to it right away, but eventually. >>>> >>>> thx. sven >>>> >>>> >>>> On Thu, Feb 22, 2018 at 12:33 PM wrote: >>>> >>>>> Hi all, >>>>> >>>>> I wanted to know, how does mmap interact with GPFS pagepool with >>>>> respect to filesystem block-size? >>>>> Does the efficiency depend on the mmap read size and the block-size of >>>>> the filesystem even if all the data is cached in pagepool? >>>>> >>>>> GPFS 4.2.3.2 and CentOS7. >>>>> >>>>> Here is what i observed: >>>>> >>>>> I was testing a user script that uses mmap to read from 100M to 500MB >>>>> files. >>>>> >>>>> The above files are stored on 3 different filesystems. >>>>> >>>>> Compute nodes - 10G pagepool and 5G seqdiscardthreshold. >>>>> >>>>> 1. 4M block size GPFS filesystem, with separate metadata and data. >>>>> Data on Near line and metadata on SSDs >>>>> 2. 1M block size GPFS filesystem as a AFM cache cluster, "with all the >>>>> required files fully cached" from the above GPFS cluster as home. Data and >>>>> Metadata together on SSDs >>>>> 3. 16M block size GPFS filesystem, with separate metadata and data. >>>>> Data on Near line and metadata on SSDs >>>>> >>>>> When i run the script first time for ?each" filesystem: >>>>> I see that GPFS reads from the files, and caches into the pagepool as >>>>> it reads, from mmdiag -- iohist >>>>> >>>>> When i run the second time, i see that there are no IO requests from >>>>> the compute node to GPFS NSD servers, which is expected since all the data >>>>> from the 3 filesystems is cached. >>>>> >>>>> However - the time taken for the script to run for the files in the 3 >>>>> different filesystems is different - although i know that they are just >>>>> "mmapping"/reading from pagepool/cache and not from disk. >>>>> >>>>> Here is the difference in time, for IO just from pagepool: >>>>> >>>>> 20s 4M block size >>>>> 15s 1M block size >>>>> 40S 16M block size. >>>>> >>>>> Why do i see a difference when trying to mmap reads from different >>>>> block-size filesystems, although i see that the IO requests are not hitting >>>>> disks and just the pagepool? >>>>> >>>>> I am willing to share the strace output and mmdiag outputs if needed. >>>>> >>>>> Thanks, >>>>> Lohit >>>>> >>>>> _______________________________________________ >>>>> gpfsug-discuss mailing list >>>>> gpfsug-discuss at spectrumscale.org >>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= End of gpfsug-discuss Digest, Vol 81, Issue 43 ********************************************** ________________________________ This message is for the recipient's use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From valleru at cbio.mskcc.org Wed Nov 20 16:46:03 2019 From: valleru at cbio.mskcc.org (valleru at cbio.mskcc.org) Date: Wed, 20 Nov 2019 11:46:03 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 81, Issue 43 In-Reply-To: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> References: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> Message-ID: <2ff249fe-15d5-4d48-88fa-12ba40d1a0f8@Spark> Hey Tushit, For us, it finally came down to the kind of applications that are doing IO. For some applications, the 16M filesystem was very bad in performance, while for some it seems to do well. Yes, I do observe a performance difference with higher pagepool but the amount of pagepool also happens to depend on the working dataset of the applications. For us - Most of the compute nodes run a variety of applications with varying datasets. Thus modifying pagepool to a large value for all nodes did not seem to be practical/enough for us. I have not made prefetchaggressivenessread/large pagepool for every node yet. I am setting it only if it is making a big difference and only for dedicated compute nodes that run only specific set of applications. I believe the issue that I was facing with respect to large block size might be because of few things such as: 1. The amount of buffers/cache that can be held in a particular amount of pagepool. 2. If the application needs to read from the same buffers multiple times or from different buffers thus deciding the need for buffer locking issue as Sven mentioned or may be not enough buffers because of large block size and not enough pagepool memory, thus needing to page in/page out and read over the network to the storage. 3. The latency to read the full block size/16M from the storage and if the application is ok with this latency. I see that most of the GPU deep learning applications are not happy with large block size latency/IOPS/reads per second. 4. Most of the scientific applications including some libraries are designed to do buffered reads (full block size) from local disk and local disk mostly has ext4 4K block size so it should not affect much when done from local disk. But with GPFS 16M filesystem - they will read 16M from the GPFS filesystem/over the network ( though they don?t need all of the data in each 16M read). Thus with prefetch, and depending on IO pattern - GPFS can fetch lots of 16M reads unnecessarily needing for the OS to discard the rest of the data once it receives. In short - I had seen considerable difference when I had modified the applications to do unbuffered reads instead of buffered reads from a large block size filesystem. 5. The large block size reads also affected the Arista 100G network switch introducing lot of network discards, and the need for a higher buffer switch or enable ECN. 6. If the application does large sequential IO reads/large random IO reads or small random IO reads from many different files. The latter applications seems to be happy with smaller block size/larger pagepool/more maxfilestocache/network switch with lesser latency. While the former applications seem to do ok with large block size. Thus we have decided to debug/strace applications one a time, and see which applications are being affected by large block size and test them with a smaller block size on a SSD/Flash backed storage. Just yesterday, I was debugging one GPU deep learning application that does lot of small random IO from thousands of different files/latency sensitive and it performed best with small block size/large pagepool and more max files to cache/prefetchaggressivenessread to zero. The reason we had moved to 16M filesystem was only because of the reason that the rotating disk storage was most happy and gives maximum throughput when on larger block size ( ignoring the latency that this might introduce for each read). However we realized that not every application would be suited for a large block size, because of the above mentioned reasons. I had also seen that those applications are usually happy from local disk, because of the smaller block size that local disk offers and the dynamic filebuffer/cache that linux offers ( unlike pagepool which is static and pinned ). The filebuffer/cache has its advantages and disadvantages because of the varying performance depending on the free memory available. 4M block size might work better than 16M for such applications, but that will also reduce the total throughput that we get from the rotating disk storage - and not give much benefit with respect to latency/IOPS. So it all depends on the applications/IO pattern, and as of now - we are in process of gathering which applications are suited for large vs smaller block size. All the above is ignoring the metadata overhead/performance. Sometimes depending on the applications - metadata performance can be a bottleneck too. Regards, Lohit On Nov 20, 2019, 10:43 AM -0500, Tushit Shukla , wrote: > Lohit, > Did you get any progress on file system?performance issue with large block size ? > Apparently we are also going through the same issue after we migrated to 16M file system (from 4MB filesystem). Only thing which gives some relief is when we increase pagepool to 32G. We also tried playing with prefetchAggressivenessRead parameter by reducing it to 1 but not sure if it is doing anything. > > thanks > From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org > Sent: Monday, October 22, 2018 12:18:58 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 81, Issue 43 > > Send gpfsug-discuss mailing list submissions to > ??????? gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > ??????? https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > or, via email, send a message with subject or body 'help' to > ??????? gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > ??????? gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > ?? 1. Re: GPFS, Pagepool and Block size -> Perfomance reduces with > ????? larger block size (Sven Oehme) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 22 Oct 2018 09:18:43 -0700 > From: Sven Oehme > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS, Pagepool and Block size -> > ??????? Perfomance reduces with larger block size > Message-ID: > ??????? > Content-Type: text/plain; charset="utf-8" > > oops, somehow that slipped my inbox, i only saw that reply right now. > > its really hard to see from a trace snipped if the lock is the issue as the > lower level locks don't show up in default traces. without having access to > source code and a detailed trace you won't make much progress here. > > sven > > > > > > On Thu, Sep 27, 2018 at 12:31 PM wrote: > > > Thank you Sven, > > > > Turning of prefetching did not improve the performance, but it did degrade > > a bit. > > > > I have made the prefetching default and took trace dump, for tracectl with > > trace=io. Let me know if you want me to paste/attach it here. > > > > May i know, how could i confirm if the below is true? > > > > 1. this could be serialization around buffer locks. as larger your > >>> blocksize gets as larger is the amount of data one of this pagepool buffers > >>> will maintain, if there is a lot of concurrency on smaller amount of data > >>> more threads potentially compete for the same buffer lock to copy stuff in > >>> and out of a particular buffer, hence things go slower compared to the same > >>> amount of data spread across more buffers, each of smaller size. > >>> > >>> > > Will the above trace help in understanding if it is a serialization issue? > > > > I had been discussing the same with GPFS support for past few months, and > > it seems to be that most of the time is being spent at cxiUXfer. They could > > not understand on why it is taking spending so much of time in cxiUXfer. I > > was seeing the same from perf top, and pagefaults. > > > > Below is snippet from what the support had said : > > > > ???????????????????????????? > > > > I searched all of the gpfsRead from trace and sort them by spending-time. > > Except 2 reads which need fetch data from nsd server, the slowest read is > > in the thread 72170. It took 112470.362 us. > > > > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum:?? 72165?????? 6.860911319 > > rdwr?????????????????? 141857.076 us + NSDIO > > > > trcrpt.2018-08-06_12.26.28.39794.lt15.trsum:?? 72170?????? 1.483947593 > > rdwr?????????????????? 112470.362 us + cxiUXfer > > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum:?? 72165?????? 6.949042593 > > rdwr??????????????????? 88126.278 us + NSDIO > > > > trcrpt.2018-08-06_12.27.03.47706.lt15.trsum:?? 72156?????? 2.919334474 > > rdwr??????????????????? 81057.657 us + cxiUXfer > > > > trcrpt.2018-08-06_12.23.30.72745.lt15.trsum:?? 72154?????? 1.167484466 > > rdwr??????????????????? 76033.488 us + cxiUXfer > > > > trcrpt.2018-08-06_12.24.06.7508.lt15.trsum:?? 72187?????? 0.685237501 > > rdwr??????????????????? 70772.326 us + cxiUXfer > > > > trcrpt.2018-08-06_12.25.17.23989.lt15.trsum:?? 72193?????? 4.757996530 > > rdwr??????????????????? 70447.838 us + cxiUXfer > > > > > > I check each of the slow IO as above, and find they all spend much time in > > the function cxiUXfer. This function is used to copy data from kernel > > buffer to user buffer. I am not sure why it took so much time. This should > > be related to the pagefaults and pgfree you observed. Below is the trace > > data for thread 72170. > > > > > >??????????????????? 1.371477231? 72170 TRACE_VNODE: gpfs_f_rdwr enter: fP > > 0xFFFF882541649400 f_flags 0x8000 flags 0x8001 op 0 iovec > > 0xFFFF881F2AFB3E70 count 1 offset 0x168F30D dentry 0xFFFF887C0CC298C0 > > private 0xFFFF883F607175C0 iP 0xFFFF8823AA3CBFC0 name '410513.svs' > > > >?????????????? .... > > > >??????????????????? 1.371483547? 72170 TRACE_KSVFS: cachedReadFast exit: > > uio_resid 16777216 code 1 err 11 > > > >?????????????? .... > > > >??????????????????? 1.371498780? 72170 TRACE_KSVFS: kSFSReadFast: oiP > > 0xFFFFC90060B46740 offset 0x168F30D dataBufP FFFFC9003645A5A8 nDesc 64 buf > > 200043C0000 valid words 64 dirty words 0 blkOff 0 > > > >??????????????????? 1.371499035? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate begin ul 0xFFFFC900333F1A40 holdCount 0 > > ioType 0x2 inProg 0x15 > > > >??????????????????? 1.371500157? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate ul 0xFFFFC900333F1A40 holdCount 0 ioType 0x2 > > inProg 0x16 err 0 > > > >??????????????????? 1.371500606? 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 toIOBuf 0 offset 6877965 len > > 9899251 > > > >??????????????????? 1.371500793? 72170 TRACE_KSVFS: cxiUXfer: ndesc 0 skip > > dataAddrP 0x200043C0000 currOffset 0 currLen 262144 bufOffset 6877965 > > > >?????????????? .... > > > >??????????????????? 1.371505949? 72170 TRACE_KSVFS: cxiUXfer: ndesc 25 skip > > dataAddrP 0x2001AF80000 currOffset 6553600 currLen 262144 bufOffset 6877965 > > > >??????????????????? 1.371506236? 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > > currOffset 6815744 tmpLen 262144 dataAddrP 0x2001AFCF30D currLen 199923 > > pageOffset 781 pageLen 3315 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.373649823? 72170 TRACE_KSVFS: cxiUXfer: nDesc 27 > > currOffset 7077888 tmpLen 262144 dataAddrP 0x20027400000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.375158799? 72170 TRACE_KSVFS: cxiUXfer: nDesc 28 > > currOffset 7340032 tmpLen 262144 dataAddrP 0x20027440000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.376661566? 72170 TRACE_KSVFS: cxiUXfer: nDesc 29 > > currOffset 7602176 tmpLen 262144 dataAddrP 0x2002C180000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.377892653? 72170 TRACE_KSVFS: cxiUXfer: nDesc 30 > > currOffset 7864320 tmpLen 262144 dataAddrP 0x2002C1C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >?????????????? .... > > > >??????????????????? 1.471389843? 72170 TRACE_KSVFS: cxiUXfer: nDesc 62 > > currOffset 16252928 tmpLen 262144 dataAddrP 0x2001D2C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.471845629? 72170 TRACE_KSVFS: cxiUXfer: nDesc 63 > > currOffset 16515072 tmpLen 262144 dataAddrP 0x2003EC80000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.472417149? 72170 TRACE_KSVFS: cxiDetachIOBuffer: > > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.472417775? 72170 TRACE_LOCK: unlock_vfs: type Data, > > key 0000000000000004:000000001B1F24BF:0000000000000001 lock_mode have ro > > token xw lock_state old [ ro:27 ] new [ ro:26 ] holdCount now 27 > > > >??????????????????? 1.472418427? 72170 TRACE_LOCK: hash tab lookup vfs: > > found cP 0xFFFFC9005FC0CDE0 holdCount now 14 > > > >??????????????????? 1.472418592? 72170 TRACE_LOCK: lock_vfs: type Data key > > 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode want ro status > > valid token xw/xw lock_state [ ro:12 ] flags 0x0 holdCount 14 > > > >??????????????????? 1.472419842? 72170 TRACE_KSVFS: kSFSReadFast: oiP > > 0xFFFFC90060B46740 offset 0x2000000 dataBufP FFFFC9003643C908 nDesc 64 buf > > 38033480000 valid words 64 dirty words 0 blkOff 0 > > > >??????????????????? 1.472420029? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate begin ul 0xFFFFC9005FC0CF98 holdCount 0 > > ioType 0x2 inProg 0xC > > > >??????????????????? 1.472420187? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate ul 0xFFFFC9005FC0CF98 holdCount 0 ioType 0x2 > > inProg 0xD err 0 > > > >??????????????????? 1.472420652? 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 toIOBuf 0 offset 0 len 6877965 > > > >??????????????????? 1.472420936? 72170 TRACE_KSVFS: cxiUXfer: nDesc 0 > > currOffset 0 tmpLen 262144 dataAddrP 0x38033480000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.472824790? 72170 TRACE_KSVFS: cxiUXfer: nDesc 1 > > currOffset 262144 tmpLen 262144 dataAddrP 0x380334C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.473243905? 72170 TRACE_KSVFS: cxiUXfer: nDesc 2 > > currOffset 524288 tmpLen 262144 dataAddrP 0x38024280000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >?????????????? .... > > > >??????????????????? 1.482949347? 72170 TRACE_KSVFS: cxiUXfer: nDesc 24 > > currOffset 6291456 tmpLen 262144 dataAddrP 0x38025E80000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483354265? 72170 TRACE_KSVFS: cxiUXfer: nDesc 25 > > currOffset 6553600 tmpLen 262144 dataAddrP 0x38025EC0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483766631? 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > > currOffset 6815744 tmpLen 262144 dataAddrP 0x38003B00000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483943894? 72170 TRACE_KSVFS: cxiDetachIOBuffer: > > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483944339? 72170 TRACE_LOCK: unlock_vfs: type Data, > > key 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode have ro > > token xw lock_state old [ ro:14 ] new [ ro:13 ] holdCount now 14 > > > >??????????????????? 1.483944683? 72170 TRACE_BRL: brUnlockM: ofP > > 0xFFFFC90069346B68 inode 455025855 snap 0 handle 0xFFFFC9003637D020 range > > 0x168F30D-0x268F30C mode ro > > > >??????????????????? 1.483944985? 72170 TRACE_KSVFS: kSFSReadFast exit: > > uio_resid 0 err 0 > > > >??????????????????? 1.483945264? 72170 TRACE_LOCK: unlock_vfs_m: type > > Inode, key 305F105B9701E60A:000000001B1F24BF:0000000000000000 lock_mode > > have ro status valid token rs lock_state old [ ro:25 ] new [ ro:24 ] > > > >??????????????????? 1.483945423? 72170 TRACE_LOCK: unlock_vfs_m: cP > > 0xFFFFC90069346B68 holdCount 25 > > > >??????????????????? 1.483945624? 72170 TRACE_VNODE: gpfsRead exit: fast err > > 0 > > > >??????????????????? 1.483946831? 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > > 0xFFFFC90035E52F78 NotQuiesced vfsOp 2 > > > >??????????????????? 1.483946975? 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > > 0xFFFFC90035E52F78 vfsOp 2 users 1-1 > > > >??????????????????? 1.483947116? 72170 TRACE_KSVFS: ReleaseDaemonSegAndSG: > > sli 38 count 2 needCleanup 0 > > > >??????????????????? 1.483947593? 72170 TRACE_VNODE: gpfs_f_rdwr exit: fP > > 0xFFFF882541649400 total_len 16777216 uio_resid 0 offset 0x268F30D rc 0 > > > > > > ??????????????????????????????????????????? > > > > > > > > Regards, > > Lohit > > > > On Sep 19, 2018, 3:11 PM -0400, Sven Oehme , wrote: > > > > the document primarily explains all performance specific knobs. general > > advice would be to longer set anything beside workerthreads, pagepool and > > filecache on 5.X systems as most other settings are no longer relevant > > (thats a client side statement) . thats is true until you hit strange > > workloads , which is why all the knobs are still there :-) > > > > sven > > > > > > On Wed, Sep 19, 2018 at 11:17 AM wrote: > > > >> Thanks Sven. > >> I will disable it completely and see how it behaves. > >> > >> Is this the presentation? > >> > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2014_UG10-5FGPFS-5FPerformance-5FSession-5Fv10.pdf&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=-Y1ncRDmgOCfWDQaXj-OcbiM5gcs0LcllAShfNapHtI&e= > >> > >> I guess i read it, but it did not strike me at this situation. I will try > >> to read it again and see if i could make use of it. > >> > >> Regards, > >> Lohit > >> > >> On Sep 19, 2018, 2:12 PM -0400, Sven Oehme , wrote: > >> > >> seem like you never read my performance presentation from a few years ago > >> ;-) > >> > >> you can control this on a per node basis , either for all i/o : > >> > >>??? prefetchAggressiveness = X > >> > >> or individual for reads or writes : > >> > >>??? prefetchAggressivenessRead = X > >>??? prefetchAggressivenessWrite = X > >> > >> for a start i would turn it off completely via : > >> > >> mmchconfig prefetchAggressiveness=0 -I -N nodename > >> > >> that will turn it off only for that node and only until you restart the > >> node. > >> then see what happens > >> > >> sven > >> > >> > >> On Wed, Sep 19, 2018 at 11:07 AM wrote: > >> > >>> Thank you Sven. > >>> > >>> I mostly think it could be 1. or some other issue. > >>> I don?t think it could be 2. , because i can replicate this issue no > >>> matter what is the size of the dataset. It happens for few files that could > >>> easily fit in the page pool too. > >>> > >>> I do see a lot more page faults for 16M compared to 1M, so it could be > >>> related to many threads trying to compete for the same buffer space. > >>> > >>> I will try to take the trace with trace=io option and see if can find > >>> something. > >>> > >>> How do i turn of prefetching? Can i turn it off for a single > >>> node/client? > >>> > >>> Regards, > >>> Lohit > >>> > >>> On Sep 18, 2018, 5:23 PM -0400, Sven Oehme , wrote: > >>> > >>> Hi, > >>> > >>> taking a trace would tell for sure, but i suspect what you might be > >>> hitting one or even multiple issues which have similar negative performance > >>> impacts but different root causes. > >>> > >>> 1. this could be serialization around buffer locks. as larger your > >>> blocksize gets as larger is the amount of data one of this pagepool buffers > >>> will maintain, if there is a lot of concurrency on smaller amount of data > >>> more threads potentially compete for the same buffer lock to copy stuff in > >>> and out of a particular buffer, hence things go slower compared to the same > >>> amount of data spread across more buffers, each of smaller size. > >>> > >>> 2. your data set is small'ish, lets say a couple of time bigger than the > >>> pagepool and you random access it with multiple threads. what will happen > >>> is that because it doesn't fit into the cache it will be read from the > >>> backend. if multiple threads hit the same 16 mb block at once with multiple > >>> 4k random reads, it will read the whole 16mb block because it thinks it > >>> will benefit from it later on out of cache, but because it fully random the > >>> same happens with the next block and the next and so on and before you get > >>> back to this block it was pushed out of the cache because of lack of enough > >>> pagepool. > >>> > >>> i could think of multiple other scenarios , which is why its so hard to > >>> accurately benchmark an application because you will design a benchmark to > >>> test an application, but it actually almost always behaves different then > >>> you think it does :-) > >>> > >>> so best is to run the real application and see under which configuration > >>> it works best. > >>> > >>> you could also take a trace with trace=io and then look at > >>> > >>> TRACE_VNOP: READ: > >>> TRACE_VNOP: WRITE: > >>> > >>> and compare them to > >>> > >>> TRACE_IO: QIO: read > >>> TRACE_IO: QIO: write > >>> > >>> and see if the numbers summed up for both are somewhat equal. if > >>> TRACE_VNOP is significant smaller than TRACE_IO you most likely do more i/o > >>> than you should and turning prefetching off might actually make things > >>> faster . > >>> > >>> keep in mind i am no longer working for IBM so all i say might be > >>> obsolete by now, i no longer have access to the one and only truth aka the > >>> source code ... but if i am wrong i am sure somebody will point this out > >>> soon ;-) > >>> > >>> sven > >>> > >>> > >>> > >>> > >>> On Tue, Sep 18, 2018 at 10:31 AM wrote: > >>> > >>>> Hello All, > >>>> > >>>> This is a continuation to the previous discussion that i had with Sven. > >>>> However against what i had mentioned previously - i realize that this > >>>> is ?not? related to mmap, and i see it when doing random freads. > >>>> > >>>> I see that block-size of the filesystem matters when reading from Page > >>>> pool. > >>>> I see a major difference in performance when compared 1M to 16M, when > >>>> doing lot of random small freads with all of the data in pagepool. > >>>> > >>>> Performance for 1M is a magnitude ?more? than the performance that i > >>>> see for 16M. > >>>> > >>>> The GPFS that we have currently is : > >>>> Version : 5.0.1-0.5 > >>>> Filesystem version: 19.01 (5.0.1.0) > >>>> Block-size : 16M > >>>> > >>>> I had made the filesystem block-size to be 16M, thinking that i would > >>>> get the most performance for both random/sequential reads from 16M than the > >>>> smaller block-sizes. > >>>> With GPFS 5.0, i made use the 1024 sub-blocks instead of 32 and thus > >>>> not loose lot of storage space even with 16M. > >>>> I had run few benchmarks and i did see that 16M was performing better > >>>> ?when hitting storage/disks? with respect to bandwidth for > >>>> random/sequential on small/large reads. > >>>> > >>>> However, with this particular workload - where it freads a chunk of > >>>> data randomly from hundreds of files -> I see that the number of > >>>> page-faults increase with block-size and actually reduce the performance. > >>>> 1M performs a lot better than 16M, and may be i will get better > >>>> performance with less than 1M. > >>>> It gives the best performance when reading from local disk, with 4K > >>>> block size filesystem. > >>>> > >>>> What i mean by performance when it comes to this workload - is not the > >>>> bandwidth but the amount of time that it takes to do each iteration/read > >>>> batch of data. > >>>> > >>>> I figure what is happening is: > >>>> fread is trying to read a full block size of 16M - which is good in a > >>>> way, when it hits the hard disk. > >>>> But the application could be using just a small part of that 16M. Thus > >>>> when randomly reading(freads) lot of data of 16M chunk size - it is page > >>>> faulting a lot more and causing the performance to drop . > >>>> I could try to make the application do read instead of freads, but i > >>>> fear that could be bad too since it might be hitting the disk with a very > >>>> small block size and that is not good. > >>>> > >>>> With the way i see things now - > >>>> I believe it could be best if the application does random reads of > >>>> 4k/1M from pagepool but some how does 16M from rotating disks. > >>>> > >>>> I don?t see any way of doing the above other than following a different > >>>> approach where i create a filesystem with a smaller block size ( 1M or less > >>>> than 1M ), on SSDs as a tier. > >>>> > >>>> May i please ask for advise, if what i am understanding/seeing is right > >>>> and the best solution possible for the above scenario. > >>>> > >>>> Regards, > >>>> Lohit > >>>> > >>>> On Apr 11, 2018, 10:36 AM -0400, Lohit Valleru , > >>>> wrote: > >>>> > >>>> Hey Sven, > >>>> > >>>> This is regarding mmap issues and GPFS. > >>>> We had discussed previously of experimenting with GPFS 5. > >>>> > >>>> I now have upgraded all of compute nodes and NSD nodes to GPFS 5.0.0.2 > >>>> > >>>> I am yet to experiment with mmap performance, but before that - I am > >>>> seeing weird hangs with GPFS 5 and I think it could be related to mmap. > >>>> > >>>> Have you seen GPFS ever hang on this syscall? > >>>> [Tue Apr 10 04:20:13 2018] [] > >>>> _ZN10gpfsNode_t8mmapLockEiiPKj+0xb5/0x140 [mmfs26] > >>>> > >>>> I see the above ,when kernel hangs and throws out a series of trace > >>>> calls. > >>>> > >>>> I somehow think the above trace is related to processes hanging on GPFS > >>>> forever. There are no errors in GPFS however. > >>>> > >>>> Also, I think the above happens only when the mmap threads go above a > >>>> particular number. > >>>> > >>>> We had faced a similar issue in 4.2.3 and it was resolved in a patch to > >>>> 4.2.3.2 . At that time , the issue happened when mmap threads go more than > >>>> worker1threads. According to the ticket - it was a mmap race condition that > >>>> GPFS was not handling well. > >>>> > >>>> I am not sure if this issue is a repeat and I am yet to isolate the > >>>> incident and test with increasing number of mmap threads. > >>>> > >>>> I am not 100 percent sure if this is related to mmap yet but just > >>>> wanted to ask you if you have seen anything like above. > >>>> > >>>> Thanks, > >>>> > >>>> Lohit > >>>> > >>>> On Feb 22, 2018, 3:59 PM -0500, Sven Oehme , wrote: > >>>> > >>>> Hi Lohit, > >>>> > >>>> i am working with ray on a mmap performance improvement right now, > >>>> which most likely has the same root cause as yours , see --> > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_2018-2DJanuary_004411.html&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=AUEL847F_a-j6Y7t1fDMj4j33vLqvI6XrrNCVS5pUyA&e= > >>>> the thread above is silent after a couple of back and rorth, but ray > >>>> and i have active communication in the background and will repost as soon > >>>> as there is something new to share. > >>>> i am happy to look at this issue after we finish with ray's workload if > >>>> there is something missing, but first let's finish his, get you try the > >>>> same fix and see if there is something missing. > >>>> > >>>> btw. if people would share their use of MMAP , what applications they > >>>> use (home grown, just use lmdb which uses mmap under the cover, etc) please > >>>> let me know so i get a better picture on how wide the usage is with GPFS. i > >>>> know a lot of the ML/DL workloads are using it, but i would like to know > >>>> what else is out there i might not think about. feel free to drop me a > >>>> personal note, i might not reply to it right away, but eventually. > >>>> > >>>> thx. sven > >>>> > >>>> > >>>> On Thu, Feb 22, 2018 at 12:33 PM wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> I wanted to know, how does mmap interact with GPFS pagepool with > >>>>> respect to filesystem block-size? > >>>>> Does the efficiency depend on the mmap read size and the block-size of > >>>>> the filesystem even if all the data is cached in pagepool? > >>>>> > >>>>> GPFS 4.2.3.2 and CentOS7. > >>>>> > >>>>> Here is what i observed: > >>>>> > >>>>> I was testing a user script that uses mmap to read from 100M to 500MB > >>>>> files. > >>>>> > >>>>> The above files are stored on 3 different filesystems. > >>>>> > >>>>> Compute nodes - 10G pagepool and 5G seqdiscardthreshold. > >>>>> > >>>>> 1. 4M block size GPFS filesystem, with separate metadata and data. > >>>>> Data on Near line and metadata on SSDs > >>>>> 2. 1M block size GPFS filesystem as a AFM cache cluster, "with all the > >>>>> required files fully cached" from the above GPFS cluster as home. Data and > >>>>> Metadata together on SSDs > >>>>> 3. 16M block size GPFS filesystem, with separate metadata and data. > >>>>> Data on Near line and metadata on SSDs > >>>>> > >>>>> When i run the script first time for ?each" filesystem: > >>>>> I see that GPFS reads from the files, and caches into the pagepool as > >>>>> it reads, from mmdiag -- iohist > >>>>> > >>>>> When i run the second time, i see that there are no IO requests from > >>>>> the compute node to GPFS NSD servers, which is expected since all the data > >>>>> from the 3 filesystems is cached. > >>>>> > >>>>> However - the time taken for the script to run for the files in the 3 > >>>>> different filesystems is different - although i know that they are just > >>>>> "mmapping"/reading from pagepool/cache and not from disk. > >>>>> > >>>>> Here is the difference in time, for IO just from pagepool: > >>>>> > >>>>> 20s 4M block size > >>>>> 15s 1M block size > >>>>> 40S 16M block size. > >>>>> > >>>>> Why do i see a difference when trying to mmap reads from different > >>>>> block-size filesystems, although i see that the IO requests are not hitting > >>>>> disks and just the pagepool? > >>>>> > >>>>> I am willing to share the strace output and mmdiag outputs if needed. > >>>>> > >>>>> Thanks, > >>>>> Lohit > >>>>> > >>>>> _______________________________________________ > >>>>> gpfsug-discuss mailing list > >>>>> gpfsug-discuss at spectrumscale.org > >>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >> > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > End of gpfsug-discuss Digest, Vol 81, Issue 43 > ********************************************** > This message is for the recipient?s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From WPeters at ATPCO.NET Wed Nov 20 18:16:23 2019 From: WPeters at ATPCO.NET (Peters, Bill) Date: Wed, 20 Nov 2019 18:16:23 +0000 Subject: [gpfsug-discuss] introduction Message-ID: Hello, The welcome email said I should introduce myself to the group. I'm Bill Peters, a Linux engineer at ATPCO (Airline Tariff Publishing Company) located in Dulles, Virginia, USA. We process airline fare data. I've was recently introduced to Spectrum Scale because we are doing a proof of concept to see if we can move some of our workload onto virtual machines running in IBM's zVM. Our x86 systems use Veritas for the network filesystem and since Veritas doesn't support the s390 arcitecture, we are using Spectrum Scale. So far it's been great, much easier to understand than Veritas. We're not doing anything too complicated. The disks are DASD on SSD. We have 3 clusters with sharing between them. At this point I expect the POC to be successful so I will probably be working with Spectrum Scale into the future. The only question I have so far is read speeds seem to be slower than Veritas. It's not slow enough to be a real problem, but if anyone has a suggestion for speeding up reads, I'd love to hear it. Other than that, I'll probably just be lurking on the email list. Thanks, -Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Nov 20 19:56:50 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 20 Nov 2019 20:56:50 +0100 Subject: [gpfsug-discuss] introduction In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Nov 20 20:06:53 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 20 Nov 2019 21:06:53 +0100 Subject: [gpfsug-discuss] introduction In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Nov 20 20:44:05 2019 From: mweil at wustl.edu (Weil, Matthew) Date: Wed, 20 Nov 2019 20:44:05 +0000 Subject: [gpfsug-discuss] fsstruct and FSCK IBM case TS002836092 Message-ID: <5889a4ef-0c62-8ee8-7347-a7bb6b3acb74@wustl.edu> Ulf Troppens, Would it be possible for you to engage and assist with this case? Thanks Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From gterryc at vmsupport.com Fri Nov 22 00:24:02 2019 From: gterryc at vmsupport.com (George Terry) Date: Thu, 21 Nov 2019 18:24:02 -0600 Subject: [gpfsug-discuss] Compatibility question Message-ID: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at googlemail.com Fri Nov 22 01:12:27 2019 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Thu, 21 Nov 2019 19:12:27 -0600 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: Message-ID: That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, wrote: > Hello, > > I have a question about compatibility between two different versions. I > have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are > clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to > application's issues I cannot update the client nodes. So, my question is, > can i have a cluster with client nodes with 3.5.0.7 version and the NSD > nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? > > Thanks for the support. > > George > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knop at us.ibm.com Fri Nov 22 04:49:12 2019 From: knop at us.ibm.com (Felipe Knop) Date: Fri, 22 Nov 2019 04:49:12 +0000 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Fri Nov 22 14:10:01 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Fri, 22 Nov 2019 14:10:01 +0000 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: Message-ID: My guess would be rhel6 clients. Though rhel6 should be able to run a 4.x release and then you are N-1. Our one and only remaining centos6 box, well we unceremoniously kicked that out of the cluster and made it use NFS. Simon From: on behalf of "luke.raimbach at googlemail.com" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 22 November 2019 at 01:12 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Compatibility question That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, > wrote: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From sadaniel at us.ibm.com Fri Nov 22 17:01:09 2019 From: sadaniel at us.ibm.com (Steven Daniels) Date: Fri, 22 Nov 2019 10:01:09 -0700 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: , Message-ID: Felipe, You are correct. The code is restricted to 4.2.2 or higher. If there is an ESS storage cluster in the environment then the recommendation I was told is 4.2.3.7 or higher. Luke, If your clients are in the same cluster then you need to be careful with the setting to minReleaseLevel, Exactly as it sounds, it will restrict membership to this level or higher. I have a client running with filesystem version 3.5.0.7 in several clusters. We just recently upgraded their ESS systems to the current release so I can verify that the file system format will carry fine. If you have ESS at 5.3.3 or higher, then the GNR recommendation is 4.2.3.7 or higher for the clients (my client can't go to v5 due to the RHEL version requirement) and they must be in a remote cluster. In my case we upgraded an ESS storage cluster to 4.1.1 to 5.0.3.3 (ESS v5.3.4.1) and for GNR we had to update this cluster to LATEST for minReleaseLevel which is 5.0.3. The client cluster is running 4.2.3.18 which is as Felipe says is N-1 and minReleaseLevel is set to 4.2.3.7 This client cluster also mounts from another storage cluster which is constrained to run at 4.1.1-8. This makes the client cluster at N+1 from its relationship with this cluster. All three clusters have minReleaseLevel set to LATEST (4.1.0, 4.2.3.7,5.0.3 , respectively). These changes would not work in a single cluster as all nodes would be restricted to the 4.1.0 or higher and the ESS would not be able to co-exist since it requires the 5.0 level. We have updated the file systems in the 4.1.1-8 cluster from 3.5.0.7 to 4.1.1 and we updated the ESS filesystems from 3.5.0.7 to 5.0.3 (compat mode) The changes all went well. Hopefully this helps you understand the nuance that is whether the clients and NSDs are in a single cluster or different clusters. The rules are slightly but importantly different. Also, when updating the filesystem versions make sure you use the "compat" flag if you have any remote clusters. If we would have run this with the "full" flag on the ESS storage cluster, it would have prevented the 4.2.3.18 client cluster from mounting its file systems. thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Felipe Knop" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 11/21/2019 09:49 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question Sent by: gpfsug-discuss-bounces at spectrumscale.org George, Coexistence either in the same cluster or across remote clusters is only supported with versions "N" and "N-1". 5.0.3 and 3.5 are "N" and "N-3", and this is not supported. If I recall, there is even logic in place that will prevent 5.x nodes from accepting connections from nodes running below 4.2 . Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 ----- Original message ----- From: Luke Raimbach Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question Date: Thu, Nov 21, 2019 8:12 PM That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, wrote: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=sHhNGNTp4CJIVbXT4t8ZvNT_mETKfXBn7XTvNDAYF7s&s=9ialbzpuZMc8DZzXtU_nqjgQ9BPsHP3DQdpHa0nKQuc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: From werner at usit.uio.no Tue Nov 26 06:25:08 2019 From: werner at usit.uio.no (Morten Werner Forsbring) Date: Tue, 26 Nov 2019 07:25:08 +0100 Subject: [gpfsug-discuss] New subscriber Message-ID: <87blsz6tob.fsf@sue.uio.no> Hi. My name is Morten Werner Forsbring and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with different services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Best regards! - Werner From helge.hauglin at usit.uio.no Tue Nov 26 10:25:21 2019 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 26 Nov 2019 11:25:21 +0100 Subject: [gpfsug-discuss] New member: Helge Hauglin Message-ID: Hi, My name is Helge Hauglin, and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Previous storage experience: NetApp and Hitachi Vantara (Hitachi NAS and block storage). Best regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From christophe.darras at atempo.com Tue Nov 26 11:21:29 2019 From: christophe.darras at atempo.com (Christophe Darras) Date: Tue, 26 Nov 2019 11:21:29 +0000 Subject: [gpfsug-discuss] New member: Helge Hauglin In-Reply-To: References: Message-ID: Hello Helge, Thanks for the introduction. Might you need some help to migrate your unstructured data from HDS and NetApp to GPFS, pls let us know, Kindest Regards, Christophe DARRAS ? Head of North Europe, Mobile : +44 7?555 99 3529 christophe.darras at atempo.com Atempo ltd N?1 EUROPEAN SOFTWARE VENDOR FOR DATA PROTECTION? ? ATEMPO.COM ? ? ? ? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Helge Hauglin Sent: mardi 26 novembre 2019 10:25 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] New member: Helge Hauglin Hi, My name is Helge Hauglin, and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Previous storage experience: NetApp and Hitachi Vantara (Hitachi NAS and block storage). Best regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From helge.hauglin at usit.uio.no Tue Nov 26 13:21:10 2019 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 26 Nov 2019 14:21:10 +0100 Subject: [gpfsug-discuss] New member: Helge Hauglin In-Reply-To: (Christophe Darras's message of "Tue, 26 Nov 2019 11:21:29 +0000") References: Message-ID: Hi Christophe, Thanks for the information, but we'll use our in-house expertise to move data to the ESS. -- Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From mark.bergman at uphs.upenn.edu Tue Nov 26 23:58:02 2019 From: mark.bergman at uphs.upenn.edu (mark.bergman at uphs.upenn.edu) Date: Tue, 26 Nov 2019 18:58:02 -0500 Subject: [gpfsug-discuss] python package installs fail due to attribute copy error (GPFS 5.0.2) Message-ID: <32070-1574812682.257155@ttRP.UhP3.tHtF> We're running GPFS 5.0.2 (DSS-G 2.2a, in the process of upgrading to 2.4b/GPFS 5.0.3.2) with NFSv4 ACLs and no explicit ACLs on the root or dependent filesets. While installing python packages within a python virtual env stored under GPFS, we get failures with "permission denied" on the destination while trying to copy attributes. The problem happens to root & end-users, and is not related to the directory permissions, ownership, or group, and is consistent and repeatable. For example, trying to use 'conda' to create a virtual environment results in the following message (severely trimmed): # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): in clone_env shutil.copystat(src, dst) in copystat _copyxattr(src, dst, follow_symlinks=follow) in _copyxattr os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) PermissionError: [Errno 13] Permission denied: '/gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6/lib/python3.5/site-packages/dm/neuralnet/__pycache__/__init__.cpython-35.pyc' `$ python/anaconda/3/bin/conda create --prefix /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6 --clone /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6` Note that the destination directory path is created, and the destination file (__init__.cython-35.pyc) is created...the exception is thrown when python tries to apply the ACLs from the source to the destination. The problem seems to be simiar to this: https://github.com/conda/conda/issues/5251 but our version of create.py is newer and already has the suggested fix. or similar to this: https://bugs.python.org/issue24538 that's affecting panfs. We've got a work-around using symlinks, but it's not something for average users. Has anyone seen this issue? Thanks, Mark "about to post to various python fora" Bergman From b.cregan at imperial.ac.uk Thu Nov 28 09:43:55 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 09:43:55 +0000 Subject: [gpfsug-discuss] Compression question Message-ID: Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 09:49:32 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 09:49:32 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Thu Nov 28 10:05:23 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 10:05:23 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 10:13:35 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 10:13:35 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 12:21:00 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 12:21:00 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Thu Nov 28 12:30:25 2019 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 28 Nov 2019 12:30:25 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 12:49:01 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 12:49:01 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191283.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191284.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191285.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Thu Nov 28 12:57:37 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 12:57:37 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , , Message-ID: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com [https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png] [https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png] [https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png] ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards [cid:15749424191280] [IBM Spectrum Scale] * * * Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com [cid:15749424191282] IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: Image.15749424191280.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: Image.15749424191281.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: Image.15749424191282.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 13:00:22 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 13:00:22 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: Message-ID: Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers > On 28. Nov 2019, at 14.57, Cregan, Bob wrote: > > ? > Hi > Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? > > After compression of an existing file we get in the snap > > -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf > file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf > metadata replication: 2 max 2 > data replication: 1 max 3 > immutable: no > appendOnly: no > flags: > storage pool name: sata1 > fileset name: userdirs > snapshot name: @GMT-2019.11.27-19.30.14 > creation time: Tue Mar 5 16:16:40 2019 > Misc attributes: ARCHIVE > Encrypted: no > > and the original file is definitely compressed. > > -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf > file name: UserGuide_13.06.pdf > metadata replication: 2 max 2 > data replication: 1 max 3 > immutable: no > appendOnly: no > flags: > storage pool name: sata1 > fileset name: userdirs > snapshot name: > creation time: Tue Mar 5 16:16:40 2019 > Misc attributes: ARCHIVE COMPRESSION (library z) > Encrypted: no > > Bob > > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger > Sent: 28 November 2019 12:30 > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Compression question > > Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial > > > Alexander, > > Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? > Daniel > > _________________________________________________________ > Daniel Kidger > IBM Technical Sales Specialist > Spectrum Scale, Spectrum NAS and IBM Cloud Object Store > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > > > > ----- Original message ----- > From: "Alexander Wolf" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 12:21 > > I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). > > Mit freundlichen Gr??en / Kind regards > > > > > > Dr. Alexander Wolf-Reber > Spectrum Scale Release Lead Architect > Department M069 / Spectrum Scale Software Development > > +49-160-90540880 > a.wolf-reber at de.ibm.com > > IBM Data Privacy Statement > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > ----- Original message ----- > From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 11:47 > > Hi > > Same principle COW. The data blocks do not get modified. > -- > Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions > Luis Bolinches > Consultant IT Specialist > Mobile Phone: +358503112585 > https://www.youracclaim.com/user/luis-bolinches > > "If you always give you will always have" -- Anonymous > > > > ----- Original message ----- > From: "Cregan, Bob" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 12:06 PM > > Just to clarify this is SS compression, so > > mmchattr --compression yes > > or an ILM equivalent > > So not a regular modification. > > Bob > > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > > > > From: Cregan, Bob > Sent: 28 November 2019 09:43 > To: gpfsug main discussion list > Subject: Compression question > > Hi All, > Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. > > What happens to the snapshot when a file is compressed in SS? The logic as I see it is > > ####### In a non compressed situation ############### > > 1) create a file, > 2) create a snapshot. > 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. > > ######In a compressed situation ############ > > 1) create a file, > 2) create a snapshot. > 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. > > You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. > Thanks > > Bob > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.cregan at imperial.ac.uk Thu Nov 28 13:05:43 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 13:05:43 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: Hi 5.0.2 with a hotfix for a lz4 compression bug Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches Sent: 28 November 2019 13:00 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Compression question Caution - This email from luis.bolinches at fi.ibm.com originated outside Imperial Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob wrote: ? Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com [https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png] [https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png] [https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png] ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards * * * Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 14:02:46 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 14:02:46 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191286.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191287.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191288.png Type: image/png Size: 1134 bytes Desc: not available URL: From luis.bolinches at fi.ibm.com Thu Nov 28 14:07:27 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 14:07:27 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: Message-ID: That is the fix. Before it was an error. So blocks that are pointed from active get compressed. Ones that are no longer linked are not modified. -- Cheers > On 28. Nov 2019, at 16.03, Alexander Wolf wrote: > > ? > I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. > > Mit freundlichen Gr??en / Kind regards > > > > > > Dr. Alexander Wolf-Reber > Spectrum Scale Release Lead Architect > Department M069 / Spectrum Scale Software Development > > +49-160-90540880 > a.wolf-reber at de.ibm.com > > IBM Data Privacy Statement > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > ----- Original message ----- > From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: "gpfsug main discussion list" > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 14:00 > > Which version are you running? > > I was involved on a big for compressed file sets and snapshots that were related to what you see. > > -- > Cheers > >> >>> On 28. Nov 2019, at 14.57, Cregan, Bob wrote: >>> >> ? >> Hi >> Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? >> >> After compression of an existing file we get in the snap >> >> -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf >> file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf >> metadata replication: 2 max 2 >> data replication: 1 max 3 >> immutable: no >> appendOnly: no >> flags: >> storage pool name: sata1 >> fileset name: userdirs >> snapshot name: @GMT-2019.11.27-19.30.14 >> creation time: Tue Mar 5 16:16:40 2019 >> Misc attributes: ARCHIVE >> Encrypted: no >> >> and the original file is definitely compressed. >> >> -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf >> file name: UserGuide_13.06.pdf >> metadata replication: 2 max 2 >> data replication: 1 max 3 >> immutable: no >> appendOnly: no >> flags: >> storage pool name: sata1 >> fileset name: userdirs >> snapshot name: >> creation time: Tue Mar 5 16:16:40 2019 >> Misc attributes: ARCHIVE COMPRESSION (library z) >> Encrypted: no >> Bob >> >> >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger >> Sent: 28 November 2019 12:30 >> To: gpfsug-discuss at spectrumscale.org >> Cc: gpfsug-discuss at spectrumscale.org >> Subject: Re: [gpfsug-discuss] Compression question >> >> Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial >> >> >> Alexander, >> >> Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? >> Daniel >> >> _________________________________________________________ >> Daniel Kidger >> IBM Technical Sales Specialist >> Spectrum Scale, Spectrum NAS and IBM Cloud Object Store >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> >> >> >> ----- Original message ----- >> From: "Alexander Wolf" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 12:21 >> >> I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). >> >> Mit freundlichen Gr??en / Kind regards >> >> >> >> >> >> >> >> >> Dr. Alexander Wolf-Reber >> Spectrum Scale Release Lead Architect >> Department M069 / Spectrum Scale Software Development >> >> +49-160-90540880 >> a.wolf-reber at de.ibm.com >> >> IBM Data Privacy Statement >> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp >> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 >> >> >> >> ----- Original message ----- >> From: "Luis Bolinches" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: gpfsug-discuss at spectrumscale.org >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 11:47 >> >> Hi >> >> Same principle COW. The data blocks do not get modified. >> -- >> Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions >> Luis Bolinches >> Consultant IT Specialist >> Mobile Phone: +358503112585 >> https://www.youracclaim.com/user/luis-bolinches >> >> "If you always give you will always have" -- Anonymous >> >> >> >> ----- Original message ----- >> From: "Cregan, Bob" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 12:06 PM >> >> Just to clarify this is SS compression, so >> >> mmchattr --compression yes >> >> or an ILM equivalent >> >> So not a regular modification. >> >> Bob >> >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> >> From: Cregan, Bob >> Sent: 28 November 2019 09:43 >> To: gpfsug main discussion list >> Subject: Compression question >> >> Hi All, >> Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. >> >> What happens to the snapshot when a file is compressed in SS? The logic as I see it is >> >> ####### In a non compressed situation ############### >> >> 1) create a file, >> 2) create a snapshot. >> 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. >> >> ######In a compressed situation ############ >> >> 1) create a file, >> 2) create a snapshot. >> 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. >> >> You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. >> Thanks >> >> Bob >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Thu Nov 28 15:00:56 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 28 Nov 2019 08:00:56 -0700 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 1134 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Thu Nov 28 15:28:28 2019 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 28 Nov 2019 15:28:28 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB17BD4DEB177B80052596E872584C0.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB18B14DEB187280052596E872584C0.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB19CD4DEB180B40052596E872584C0.png Type: image/png Size: 1134 bytes Desc: not available URL: From scale at us.ibm.com Thu Nov 28 17:09:37 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 28 Nov 2019 22:39:37 +0530 Subject: [gpfsug-discuss] =?utf-8?q?python_package_installs_fail_due_to_at?= =?utf-8?q?tribute_copy=09error_=28GPFS_5=2E0=2E2=29?= In-Reply-To: <32070-1574812682.257155@ttRP.UhP3.tHtF> References: <32070-1574812682.257155@ttRP.UhP3.tHtF> Message-ID: Hi Diane, Can you please help customer with the below issue. Or else can you point me to the right folks who can help here. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: mark.bergman at uphs.upenn.edu To: gpfsug main discussion list Date: 27-11-2019 05:47 Subject: [EXTERNAL] [gpfsug-discuss] python package installs fail due to attribute copy error (GPFS 5.0.2) Sent by: gpfsug-discuss-bounces at spectrumscale.org We're running GPFS 5.0.2 (DSS-G 2.2a, in the process of upgrading to 2.4b/GPFS 5.0.3.2) with NFSv4 ACLs and no explicit ACLs on the root or dependent filesets. While installing python packages within a python virtual env stored under GPFS, we get failures with "permission denied" on the destination while trying to copy attributes. The problem happens to root & end-users, and is not related to the directory permissions, ownership, or group, and is consistent and repeatable. For example, trying to use 'conda' to create a virtual environment results in the following message (severely trimmed): # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): in clone_env shutil.copystat(src, dst) in copystat _copyxattr(src, dst, follow_symlinks=follow) in _copyxattr os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) PermissionError: [Errno 13] Permission denied: '/gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6/lib/python3.5/site-packages/dm/neuralnet/__pycache__/__init__.cpython-35.pyc' `$ python/anaconda/3/bin/conda create --prefix /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6 --clone /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6` Note that the destination directory path is created, and the destination file (__init__.cython-35.pyc) is created...the exception is thrown when python tries to apply the ACLs from the source to the destination. The problem seems to be simiar to this: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda_conda_issues_5251&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=8cqECGE7aQzVZwSOUDWjUbgMiyWLzpVqThnCrhMXwGc&e= but our version of create.py is newer and already has the suggested fix. or similar to this: https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue24538&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=Zyfy_KMaSuhXSmm5sUTBMiRqEiaIfD4BLSyRfpNjQkY&e= that's affecting panfs. We've got a work-around using symlinks, but it's not something for average users. Has anyone seen this issue? Thanks, Mark "about to post to various python fora" Bergman _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=uVXOGz3UqVUU4d3Ov4e4-8wLRnxTsNtRPOv_iWcmOjM&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From scale at us.ibm.com Thu Nov 28 17:22:43 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 28 Nov 2019 22:52:43 +0530 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: Hai Zhong, Can you please help the customer with their compression related query. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Daniel Kidger" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 28-11-2019 20:58 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). And likewise for deeper (older) snapshots. sDaniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Olaf Weiser" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA (of course.. only as long nobody changed the file since the snapshot was taken) From: "Alexander Wolf" To: gpfsug-discuss at spectrumscale.org Date: 11/28/2019 07:03 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Gr??en / Kind regards IBM Spectrum Scale Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug main discussion list" Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 14:00 Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob wrote: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question |------------------------------------------------------------------------------| |Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial| | | |------------------------------------------------------------------------------| Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=6F5tNsgC3a8tnTmVCVxGFJgNmWLUYm_MUO-zYDgeQ5o&s=F1aJcNbQxqosWD5WimhGS19MpeszOyIp9CxuUib-c2Q&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14065652.gif Type: image/gif Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14488338.gif Type: image/gif Size: 6645 bytes Desc: not available URL: From zhouhz at cn.ibm.com Fri Nov 29 01:55:31 2019 From: zhouhz at cn.ibm.com (Hai Zhong HZ Zhou) Date: Fri, 29 Nov 2019 01:55:31 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361973.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361974.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361975.png Type: image/png Size: 1134 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Fri Nov 29 07:22:56 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Fri, 29 Nov 2019 07:22:56 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , Message-ID: Hi Thanks very much everyone for this. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Hai Zhong HZ Zhou Sent: 29 November 2019 01:55 To: IBM Spectrum Scale Cc: gpfsug-discuss-bounces at spectrumscale.org ; gpfsug main discussion list ; Leo Luan Subject: Re: [gpfsug-discuss] Compression question Caution - This email from zhouhz at cn.ibm.com originated outside Imperial The statement from Olaf and Alex in below emails are correct. Firstly, compressing and decompressing files in active file system doesn't not trigger data blocks copy-on-write, that is just deallocating the unnecessary original data blocks and put compressed data in few data blocks, and no data blocks copy to snapshot. Secondly, when reading the file from snapshot, it will be redirected to active file system because there's no data blocks in snapshot, and then doing decompression in-memory for snapshot read, while the data on disk is still kept compressed. Regards, Hai Zhong Zhou -----Huzefa H Pancha/India/IBM wrote: ----- To: Hai Zhong HZ Zhou/China/IBM at IBMCN From: IBM Spectrum Scale/Poughkeepsie/IBM Sent by: Huzefa H Pancha/India/IBM Date: 11/29/2019 02:33AM Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list >, Leo Luan/Almaden/IBM at IBMUS Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] Compression question Hai Zhong, Can you please help the customer with their compression related query. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Daniel Kidger" ---28-11-2019 20:58:12---Olaf's explanation makes excellent sense. So is this defi]"Daniel Kidger" ---28-11-2019 20:58:12---Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshot From: "Daniel Kidger" > To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 28-11-2019 20:58 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). And likewise for deeper (older) snapshots. sDaniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Olaf Weiser" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA (of course.. only as long nobody changed the file since the snapshot was taken) From: "Alexander Wolf" > To: gpfsug-discuss at spectrumscale.org Date: 11/28/2019 07:03 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Gr??en / Kind regards [cid:1574992361973] [IBM Spectrum Scale] Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com [cid:1574992361975] IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug main discussion list" > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 14:00 Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob > wrote: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Daniel Kidger > Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Alexander Wolf" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list > Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361973.png Type: image/png Size: 1134 bytes Desc: Image.1574992361973.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361974.png Type: image/png Size: 6645 bytes Desc: Image.1574992361974.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361975.png Type: image/png Size: 1134 bytes Desc: Image.1574992361975.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From kraemerf at de.ibm.com Fri Nov 29 08:35:22 2019 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Fri, 29 Nov 2019 09:35:22 +0100 Subject: [gpfsug-discuss] FYI - Nvidia Magnum IO SC19 Accelerating data for NVIDIA GPUs with IBM Spectrum Scale Message-ID: Nvidia Magnum IO SC19 Accelerating data for NVIDIA GPUs with IBM Spectrum Scale NVIDIA announced the Magnum IO solution that complements leading-edge data management systems such as IBM Spectrum Scale and helps address AI and big data analytics I/O challenges. https://news.developer.nvidia.com/magnum-io-software-suite/ https://youtu.be/_iIAEflTZuE Accelerating data for NVIDIA GPUs - IBM IT Infrastructure Blog https://www.ibm.com/blogs/systems/accelerating-data-for-nvidia-gpus/ -frank- P.S. Oracle Cloud is now using Spectrum Scale :-) https://blogs.oracle.com/cloud-infrastructure/the-fastest-file-servers-in-the-public-cloud P.S. and the winner is ?! CM> "Coldago is a relatively little-known analyst firm and its output stands up well against prominent research firms such as Gartner and Forrester." https://blocksandfiles.com/2019/11/28/coldago-research-26-file-storage-suppliers/ Frank Kraemer IBM Consulting IT Specialist / Client Technical Architect Am Weiher 24, 65451 Kelsterbach, Germany mailto:kraemerf at de.ibm.com Mobile +49171-3043699 IBM Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinto at scinet.utoronto.ca Fri Nov 1 23:40:31 2019 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Fri, 1 Nov 2019 19:40:31 -0400 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= Message-ID: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> How can I force secondary processes to use the folder instructed by the -g option? I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 -P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From spectrumscale at kiranghag.com Sat Nov 2 03:35:32 2019 From: spectrumscale at kiranghag.com (KG) Date: Sat, 2 Nov 2019 09:05:32 +0530 Subject: [gpfsug-discuss] Windows + LDAP + Linux clients interop Message-ID: Hi Folks I need to integrate Windows 10 clients into Spectrum Scale cluster that will have Linux clients. - The Linux clients get authenticated via LDAP - The Windows 10 clients authenticate via AD - There would not be any folder with simultaneous Win/Lin access however the ACLS are required to protect file/directory access amongst users I would like to know how file permissions and overall interop would happen. Do I need to change Win10 client to authenticate via LDAP? OR can they use LDAP while connecting and using GPFS mounts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From makaplan at us.ibm.com Mon Nov 4 01:24:35 2019 From: makaplan at us.ibm.com (Marc A Kaplan) Date: Sun, 3 Nov 2019 20:24:35 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= In-Reply-To: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> References: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> Message-ID: Please show us the 2 or 3 mmbackup commands that you would like to run concurrently. Peeking into the script, I find: if [[ $scope == "inode-space" ]] then deviceSuffix="${deviceName}.${filesetName}" else deviceSuffix="${deviceName}" I believe mmbackup is designed to allow concurrent backup of different independent filesets within the same filesystem, Or different filesystems... And a single mmbackup instance can drive several TSM servers, which can be named with an option or in the dsm.sys file: # --tsm-servers TSMserver[,TSMserver...] # List of TSM servers to use instead of the servers in the dsm.sys file. From: Jaime Pinto To: gpfsug main discussion list Date: 11/01/2019 07:40 PM Subject: [EXTERNAL] [gpfsug-discuss] mmbackup ?g GlobalWorkDirectory not being followed Sent by: gpfsug-discuss-bounces at spectrumscale.org How can I force secondary processes to use the folder instructed by the -g option? I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 -P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. Thanks Jaime ************************************ TELL US ABOUT YOUR SUCCESS STORIES https://urldefense.proofpoint.com/v2/url?u=http-3A__www.scinethpc.ca_testimonials&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8D_-g0uvUjmBxM3FoCM3Jru71qum_AlcQYOe2gaC9Iw&s=rKNspohw_8ulLhO5Epvqly_vRyiBLxylBWPNKkea2eU&e= ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=cvpnBBH0j41aQy0RPiG2xRL_M8mTc1izuQD3_PmtjZ8&m=8D_-g0uvUjmBxM3FoCM3Jru71qum_AlcQYOe2gaC9Iw&s=Yahjw-3p5PGqhgawsVqB2LuwZ151Fov398camDm4XwY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From pinto at scinet.utoronto.ca Mon Nov 4 01:56:35 2019 From: pinto at scinet.utoronto.ca (Jaime Pinto) Date: Sun, 3 Nov 2019 20:56:35 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmbackup_=E2=80=90g_GlobalWorkDirector?= =?utf-8?q?y_not_being_followed?= In-Reply-To: References: <15ae3e3d-9274-13a1-06e0-9ddea4f200a7@scinet.utoronto.ca> Message-ID: <5bf94749-4add-c4f4-63df-21551c5111e1@scinet.utoronto.ca> On 11/3/2019 20:24:35, Marc A Kaplan wrote: > Please show us the 2 or 3 mmbackup commands that you would like to run concurrently. Hey Marc, They would be pretty similar, with the only different being the target TSM server, determined by sourcing a different dsmenv1(2 or 3) prior to the start of each instance, each with its own dsm.sys (3 wrappers). (source dsmenv1; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg1 --scope inodespace -v -a 8 -L 2) (source dsmenv3; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg2 --scope inodespace -v -a 8 -L 2) (source dsmenv3; /usr/lpp/mmfs/bin/mmbackup /gpfs/fs1/home -N tapenode3-ib -s /dev/shm --tsm-errorlog $tmpDir/home-tsm-errorlog -g /gpfs/fs1/home/.mmbackupCfg3 --scope inodespace -v -a 8 -L 2) I was playing with the -L (to control the policy), but you bring up a very good point I had not experimented with, such as a single traverse for multiple target servers. It may be just what I need. I'll try this next. Thank you very much, Jaime > > Peeking into the script, I find: > > if [[ $scope == "inode-space" ]] > then > deviceSuffix="${deviceName}.${filesetName}" > else > deviceSuffix="${deviceName}" > > > I believe mmbackup is designed to allow concurrent backup of different independent filesets within the same filesystem, Or different filesystems... > > And a single mmbackup instance can drive several TSM servers, which can be named with an option or in the dsm.sys file: > > # --tsm-servers TSMserver[,TSMserver...] > # List of TSM servers to use instead of the servers in the dsm.sys file. > > > > Inactive hide details for Jaime Pinto ---11/01/2019 07:40:47 PM---How can I force secondary processes to use the folder instrucJaime Pinto ---11/01/2019 07:40:47 PM---How can I force secondary processes to use the folder instructed by the -g option? I started a mmbac > > From: Jaime Pinto > To: gpfsug main discussion list > Date: 11/01/2019 07:40 PM > Subject: [EXTERNAL] [gpfsug-discuss] mmbackup ?g GlobalWorkDirectory not being followed > Sent by: gpfsug-discuss-bounces at spectrumscale.org > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > > > How can I force secondary processes to use the folder instructed by the -g option? > > I started a mmbackup with ?g /gpfs/fs1/home/.mmbackupCfg1 and another with ?g /gpfs/fs1/home/.mmbackupCfg2 (and another with ?g /gpfs/fs1/home/.mmbackupCfg3 ...) > > However I'm still seeing transient files being worked into a "/gpfs/fs1/home/.mmbackupCfg" folder (created by magic !!!). This absolutely can not happen, since it's mixing up workfiles from multiple mmbackup instances for different target TSM servers. > > See below the "-f /gpfs/fs1/home/.mmbackupCfg/prepFiles" created by mmapplypolicy (forked by mmbackup): > > DEBUGtsbackup33: /usr/lpp/mmfs/bin/mmapplypolicy "/gpfs/fs1/home" -g /gpfs/fs1/home/.mmbackupCfg2 -N tapenode3-ib -s /dev/shm -L 2 --qos maintenance -a 8 ?-P /var/mmfs/mmbackup/.mmbackupRules.fs1.home -I prepare -f /gpfs/fs1/home/.mmbackupCfg/prepFiles --irule0 --sort-buffer-size=5% --scope inodespace > > > Basically, I don't want a "/gpfs/fs1/home/.mmbackupCfg" folder to ever exist. Otherwise I'll be forced to serialize these backups, to avoid the different mmbackup instances tripping over each other. The serializing is very undesirable. > > Thanks > Jaime > > > ************************************ TELL US ABOUT YOUR SUCCESS STORIES http://www.scinethpc.ca/testimonials ************************************ --- Jaime Pinto - Storage Analyst SciNet HPC Consortium - Compute/Calcul Canada www.scinet.utoronto.ca - www.computecanada.ca University of Toronto 661 University Ave. (MaRS), Suite 1140 Toronto, ON, M5G1M1 P: 416-978-2755 C: 416-505-1477 From heinrich.billich at id.ethz.ch Mon Nov 4 17:13:50 2019 From: heinrich.billich at id.ethz.ch (Billich Heinrich Rainer (ID SD)) Date: Mon, 4 Nov 2019 17:13:50 +0000 Subject: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Message-ID: Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner -------------- next part -------------- An HTML attachment was scrubbed... URL: From YARD at il.ibm.com Mon Nov 4 19:21:26 2019 From: YARD at il.ibm.com (Yaron Daniel) Date: Mon, 4 Nov 2019 21:21:26 +0200 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=Mg9Yxn6axX4-iDSDZNIhF58cDheSv41MfR8uVXS_q58&s=bwXBF_0oFwkRREwv9IUvhXGgbQtjhEnJR5Xma7_XFIU&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From sadaniel at us.ibm.com Mon Nov 4 19:43:18 2019 From: sadaniel at us.ibm.com (Steven Daniels) Date: Mon, 4 Nov 2019 12:43:18 -0700 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=LfJIsA_kqUNu0c81SWoB1avoQI9WhsKRAILIZcOW8ts&s=0CsyJjJK2f21MXg1WtTCrGNLrg31sqbejXCMQzLyNQo&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From YARD at il.ibm.com Tue Nov 5 11:42:34 2019 From: YARD at il.ibm.com (Yaron Daniel) Date: Tue, 5 Nov 2019 13:42:34 +0200 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: Hi Steven Can you please try to run mmlsnsd - and let me know if you can change the nsd name before/after configure them with mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Steven Daniels" To: gpfsug main discussion list Date: 04/11/2019 21:44 Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Bn1XE9uK2a9CZQ8qKnJE3Q&m=EzTElY1rKKm1TuasJlFc6kJOA5qcwp0FPH71ofd8CsA&s=7Tbbpw_RbqYWzzoaTVvo7V_11FP9ytTQRHw_TWgC24Q&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From scale at us.ibm.com Tue Nov 5 13:51:45 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 5 Nov 2019 08:51:45 -0500 Subject: [gpfsug-discuss] =?utf-8?q?mmvdisk_-_how_to_see_which_recovery_gr?= =?utf-8?q?oups_are=09managed_by_mmvdisk=3F?= In-Reply-To: References: Message-ID: As far as I know there is no way to change nsd names once they have been created, and yes mmvdisk automatically generates the nsd names. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479 . If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/05/2019 06:42 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi Steven Can you please try to run mmlsnsd - and let me know if you can change the nsd name before/after configure them with mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Steven Daniels" To: gpfsug main discussion list Date: 04/11/2019 21:44 Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org the vdiskset names can be renamed - https://www.ibm.com/support/knowledgecenter/en/SSYSP8_5.3.1/com.ibm.spectrum.scale.raid.v5r01.adm.doc/bl8adm_mmvdisk_vdiskset.htm Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Yaron Daniel" To: gpfsug main discussion list Date: 11/04/2019 12:21 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi From what i know - there is no control on the NSD names - they got automatically generated by mmvdisk. Regards Yaron Daniel 94 Em Ha'Moshavot Rd Storage Architect ? IL Lab Services (Storage) Petach Tiqva, 49527 IBM Global Markets, Systems HW Sales Israel Phone: +972-3-916-5672 Fax: +972-3-916-5672 Mobile: +972-52-8395593 e-mail: yard at il.ibm.com Webex: https://ibm.webex.com/meet/yard IBM Israel From: "Billich Heinrich Rainer (ID SD)" To: gpfsug main discussion list Date: 04/11/2019 19:20 Subject: [EXTERNAL] [gpfsug-discuss] mmvdisk - how to see which recovery groups are managed by mmvdisk? Sent by: gpfsug-discuss-bounces at spectrumscale.org Hello, I try to get acquainted with mmvdisk: can I decide on the names of vdisks/nsds which mmvdisk creates? Tools like mmdf still show nsd devices, no vdisk-sets, hence a proper naming helps. RG001VS001 isn?t always what I would choose. Of course I can just not use mmvdisk where possible, but some recoverygroups already got migrated, so I have to. And I admit I like the output of ?# mmvdisk recoverygroup list --declustered-array? which gives a nice. Summary of total/free disk space. Cheers, Heiner _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=JZEHO0Ia1aoBD_GuPcVjZWgvVTYFsFL5YXqg-0nZCbw&s=iN3GzQg9lDBGVOeJR99XKrc7spI5ZQWGat6gNX5bN8s&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 1114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3847 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4266 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3747 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3793 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4301 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3739 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 3855 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4338 bytes Desc: not available URL: From Robert.Oesterlin at nuance.com Tue Nov 5 18:04:30 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Tue, 5 Nov 2019 18:04:30 +0000 Subject: [gpfsug-discuss] Reminder: Sign up for the SSUG meeting at SC19 if you plan to attend Message-ID: Reminder for all those going to SC19 and plan on attending the Spectrum Scale User Group meeting on Sunday November 17th - please pre-register! Agenda details and registration link is here: https://www.spectrumscaleug.org/event/spectrum-scale-user-group-meeting-sc19/ Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Wed Nov 6 13:06:56 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Wed, 6 Nov 2019 13:06:56 +0000 Subject: [gpfsug-discuss] CIUK Call for Speakers Message-ID: Hi All, It?s just a month until CIUK and as in the past we?re running a Spectrum Scale user group as part of the conference. We?ll be running in the morning of 5th December and are looking for a couple of speakers who could do user talks for the meeting. Please let me know if you are interested in doing a site update/talk on something there! As usual, for this user group you must be a registered CIUK attendee. Thanks Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From damir.krstic at gmail.com Fri Nov 8 16:25:24 2019 From: damir.krstic at gmail.com (Damir Krstic) Date: Fri, 8 Nov 2019 10:25:24 -0600 Subject: [gpfsug-discuss] gpfs client and turning off swap Message-ID: I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. Let me know when you get a chance. Thank you. Damir -------------- next part -------------- An HTML attachment was scrubbed... URL: From david_johnson at brown.edu Fri Nov 8 16:31:20 2019 From: david_johnson at brown.edu (david_johnson at brown.edu) Date: Fri, 8 Nov 2019 11:31:20 -0500 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: References: Message-ID: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> We have most of our clients network booted and diskless ? no swap possible. Gpfs still works until someone runs the node out of memory.... -- ddj Dave Johnson > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > ? > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > Let me know when you get a chance. > > Thank you. > Damir > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From damir.krstic at gmail.com Fri Nov 8 16:37:03 2019 From: damir.krstic at gmail.com (Damir Krstic) Date: Fri, 8 Nov 2019 10:37:03 -0600 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: Thanks - that's what I thought. We also (many refreshes ago) ran diskless images with RedHat 6 and GPFS 3.4 and ran into no issues with having swap off. I figured to ask in case something has changed with the Spectrum Scale. Damir On Fri, Nov 8, 2019 at 10:31 AM wrote: > We have most of our clients network booted and diskless ? no swap > possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic > wrote: > > > > ? > > I was wondering if it's safe to turn off swap on gpfs client machines? > we have a case where checkpointing is swapping and we would like to prevent > it from doing so by disabling swap. However, the gpfs manual admin. manual > states to have swap enabled and sufficiently large, but it does not > distinguish between clients and IO servers. > > > > Let me know when you get a chance. > > > > Thank you. > > Damir > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Fri Nov 8 16:44:47 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Fri, 8 Nov 2019 16:44:47 +0000 Subject: [gpfsug-discuss] SSUG meeting at SC19 - Important Updates! Message-ID: <6BEA6CAB-A84E-4C1E-8A56-F600F04C4082@nuance.com> For those of you going to (or planning on going) to the User Group meeting at SC19, here is some important updates: You should pre-register here: https://www.spectrumscaleug.org/event/spectrum-scale-user-group-meeting-sc19/ Box Lunches: Thanks to the generous supports of Mark III Systems and Lenovo, we will have box lunches available. These should arrive about 11:30AM on Sunday 11/17. There is a limited number (100) and will include a variety of sandwiches, and some vegetarian selections. Drinks are NOT included. First come, first serve. These are free of charge, and are available to any of those attending either the morning or afternoon session(s). Wi-Fi: Despite the best efforts of the Principals, Sponsors, and IBM, we haven?t been able to make this work. There WILL be a cable drop to the podium for presentations, but no public Wi-Fi access - Hotel Guest Wi-Fi should be available. Looking forward to seeing everyone at SC19! Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From novosirj at rutgers.edu Fri Nov 8 16:49:21 2019 From: novosirj at rutgers.edu (Ryan Novosielski) Date: Fri, 8 Nov 2019 16:49:21 +0000 Subject: [gpfsug-discuss] GPFS diskless/stateless clients (was: Re: gpfs client and turning off swap) In-Reply-To: References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: <229B3495-6D3D-40DF-8927-0960F751749C@rutgers.edu> We run stateless machines today with CentOS 7.x and have run GPFS 4.1.0.8, 4.2.3.x, and 5.0.x. We don?t use swap either in most cases, and have never needed to do anything special for that. We?ve generally needed to rig ExecStartPre stuff in systemd in order to make sure that the machines could restore their cluster membership. If anyone else can share how they do that, it might be interesting information. In our case, this works (our provisioning system writes these variables as the actual host names) ? we used to put this in an override to gpfs.service; I believe mmautoload.service is relatively new (either in 4.2.3.x to 5.0.x or 4.1.x to 4.2.x). [root at test mmautoload.service.d]# pwd /etc/systemd/system/mmautoload.service.d [root at test mmautoload.service.d]# cat override.conf [Unit] Before=slurmd.service After=sys-subsystem-net-devices-ib0.device [Service] ExecStartPre=/usr/lpp/mmfs/bin/mmsdrrestore -p $CLUSTERMGR -R /usr/bin/scp ExecStartPre=/usr/lpp/mmfs/bin/mmauth genkey propagate -N $NODENAME -- ____ || \\UTGERS, |---------------------------*O*--------------------------- ||_// the State | Ryan Novosielski - novosirj at rutgers.edu || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus || \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark `' > On Nov 8, 2019, at 11:37 AM, Damir Krstic wrote: > > Thanks - that's what I thought. We also (many refreshes ago) ran diskless images with RedHat 6 and GPFS 3.4 and ran into no issues with having swap off. I figured to ask in case something has changed with the Spectrum Scale. > Damir > > On Fri, Nov 8, 2019 at 10:31 AM wrote: > We have most of our clients network booted and diskless ? no swap possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > > > ? > > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > > > Let me know when you get a chance. > > > > Thank you. > > Damir > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss From skylar2 at uw.edu Fri Nov 8 16:52:33 2019 From: skylar2 at uw.edu (Skylar Thompson) Date: Fri, 8 Nov 2019 16:52:33 +0000 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> Message-ID: <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> We don't run diskless systems, but use cgroups where possible to limit the damage users can do to a node. We derate the node's total usable memory by 2GB (OS) + GPFS pagepool to avoid paging whenever possible. On Fri, Nov 08, 2019 at 11:31:20AM -0500, david_johnson at brown.edu wrote: > We have most of our clients network booted and diskless ??? no swap possible. Gpfs still works until someone runs the node out of memory.... > > -- ddj > Dave Johnson > > > On Nov 8, 2019, at 11:25 AM, Damir Krstic wrote: > > > > ??? > > I was wondering if it's safe to turn off swap on gpfs client machines? we have a case where checkpointing is swapping and we would like to prevent it from doing so by disabling swap. However, the gpfs manual admin. manual states to have swap enabled and sufficiently large, but it does not distinguish between clients and IO servers. > > > > Let me know when you get a chance. -- -- Skylar Thompson (skylar2 at u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine From valdis.kletnieks at vt.edu Fri Nov 8 21:15:12 2019 From: valdis.kletnieks at vt.edu (Valdis Kl=?utf-8?Q?=c4=93?=tnieks) Date: Fri, 08 Nov 2019 16:15:12 -0500 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: References: Message-ID: <30980.1573247712@turing-police> On Fri, 08 Nov 2019 10:25:24 -0600, Damir Krstic said: > I was wondering if it's safe to turn off swap on gpfs client machines? we > have a case where checkpointing is swapping and we would like to prevent it > from doing so by disabling swap. However, the gpfs manual admin. GPFS will work just fine without a swap space if the system has sufficient RAM. However, if checkpointing is swapping, you don't have enough RAM (at least with your current configuration), and turning off swap will result in processes being killed due to lack of memory. You *might* get it to work by tuning system config variables to reduce RAM consumption. But that's better tested while you still have swap defined, and not removing swap until you have a system not needing it. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 832 bytes Desc: not available URL: From jonathan.buzzard at strath.ac.uk Sat Nov 9 09:27:10 2019 From: jonathan.buzzard at strath.ac.uk (Jonathan Buzzard) Date: Sat, 9 Nov 2019 09:27:10 +0000 Subject: [gpfsug-discuss] gpfs client and turning off swap In-Reply-To: <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> References: <0E610514-7806-4EB9-A336-28A992B5B495@brown.edu> <20191108165233.mp6fzgbvem4d6jhl@utumno.gs.washington.edu> Message-ID: On 08/11/2019 16:52, Skylar Thompson wrote: > We don't run diskless systems, but use cgroups where possible to limit the > damage users can do to a node. We derate the node's total usable memory by > 2GB (OS) + GPFS pagepool to avoid paging whenever possible. > I would add that as the RAM in the node rises swap becomes pointless. We find that it's a bit pointless at 192GB of RAM in a node, for our large memory nodes which have 3TB of RAM, swap is utterly pointless. In fact there is more RAM than there is disk space. In either case the old rules about setting swap to be twice RAM or even swap equals RAM are either hard to achieve to impossible. JAB. -- Jonathan A. Buzzard Tel: +44141-5483420 HPC System Administrator, ARCHIE-WeSt. University of Strathclyde, John Anderson Building, Glasgow. G4 0NG From julian.jakobs at cec.mpg.de Mon Nov 11 11:25:29 2019 From: julian.jakobs at cec.mpg.de (Jakobs, Julian) Date: Mon, 11 Nov 2019 11:25:29 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 Message-ID: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Hello, has anyone already tried Spectrum Scale on RHEL 8.1? I can see from the GPFS FAQ that "RHEL 8" (no minor version indicated) is supported as of the 5.0.4 release. However latest kernel level fully tested is 4.18.0-80, indicating that only RHEL 8.0 was tested. I tested an installation with 8.1 (kernel version 4.18.0-147) and mmbuildgpl failed due to errors while compiling the gpl (incompatible pointer type). Is this expected behaviour or is there maybe something else wrong with the installation? If this needs a new GPFS release, is there an estimated time? I would much prefer to install it with RHEL 8.1 due to 8.0 not being a EUS release. Best, Julian Jakobs -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6220 bytes Desc: not available URL: From stockf at us.ibm.com Mon Nov 11 12:47:28 2019 From: stockf at us.ibm.com (Frederick Stock) Date: Mon, 11 Nov 2019 12:47:28 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> References: <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: From Robert.Oesterlin at nuance.com Mon Nov 11 13:25:27 2019 From: Robert.Oesterlin at nuance.com (Oesterlin, Robert) Date: Mon, 11 Nov 2019 13:25:27 +0000 Subject: [gpfsug-discuss] Force removal of problematic inode? Message-ID: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> Any ideas on how I could get out of this, short of a full mmfsck? Directory paths shortened to obfuscate data. [root at nrg5-gpfs01 remove]# tsfindinode -i 11198927 /gpfs/lm gpfs_iopen(/gpfs/lm/.../build_respell_map)(184769551:0xD4F8): Device or resource busy ^C ls -li /gpfs/lm/?/ ls: cannot access /gpfs/lm/.../build_respell_map: Device or resource busy total 0 ? d????????? ? ? ? ? ? build_respell_map [root at nrg5-gpfs01 remove]# rm -rf /gpfs/lm/.../* rm: cannot remove ?/gpfs/lm/.../build_respell_map?: Device or resource busy Bob Oesterlin Sr Principal Storage Engineer, Nuance -------------- next part -------------- An HTML attachment was scrubbed... URL: From stockf at us.ibm.com Mon Nov 11 13:28:19 2019 From: stockf at us.ibm.com (Frederick Stock) Date: Mon, 11 Nov 2019 13:28:19 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: References: , <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: From A.Wolf-Reber at de.ibm.com Mon Nov 11 13:30:09 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Mon, 11 Nov 2019 13:30:09 +0000 Subject: [gpfsug-discuss] GPFS on RHEL 8.1 In-Reply-To: References: , <359f4373f7a04bf3a37ee370aa59ccbb@cec.mpg.de> Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746414.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746415.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.157345930746416.png Type: image/png Size: 1134 bytes Desc: not available URL: From alexander.zimmermann at id.ethz.ch Mon Nov 11 14:16:34 2019 From: alexander.zimmermann at id.ethz.ch (Zimmermann Alexander (ID SD)) Date: Mon, 11 Nov 2019 14:16:34 +0000 Subject: [gpfsug-discuss] orphaned deleted fileset Message-ID: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> Hi, we've an orphaned deleted fileset in one of our filesystems: [azi4ea at nas22ems01 ~]$ mmlsfileset /dev/fs2201 --deleted Filesets in file system 'fs2201': Name Status Path (chab_services_admin_s1) Deleted /fs2201/.snapshots/@GMT-2019.10.21-10.45.55/chab_services_admin_s1 (biol_bc_azzalin_1) Deleted -- [azi4ea at nas22ems01 ~]$ biol_bc_azzalin_1 is now in that state for roughly 1 week, initially it was in the state Deleting until I ran (another?) mmdelfileset command on it. The order to delete it in our automation tool was completed in July so it's hard find out what went wrong back then. Will it get cleaned up on its own and I'm only too impatient or do I need to intervene to get rid of it completely? Many thanks in advance for your aid! Kind regards, Alexander Zimmermann Alexander Zimmermann ETH Z?rich ID Systemdienste - Speicher Weinbergstrasse 11, WEC C16 8092 Z?rich Tel.: +41 44 632 72 39 E-Mail: alexander.zimmermann at id.ethz.ch www.id.ethz.ch -------------- next part -------------- An HTML attachment was scrubbed... URL: From will.schmied at stjude.org Mon Nov 11 22:02:19 2019 From: will.schmied at stjude.org (Schmied, Will) Date: Mon, 11 Nov 2019 22:02:19 +0000 Subject: [gpfsug-discuss] Job opportunity: HPC Storage Architect at St. Jude Message-ID: <1E710D36-37F5-4BCC-BE23-8A3A192B67BA@stjude.org> Hello everyone, St. Jude Children?s Research Hospital (Memphis, TN) has recently posted a job opening for a HPC Storage Architect, a senior level position working primarily to operate and maintain multiple Spectrum Scale clusters in support of research and other HPC workloads. You can view the job posting, and begin your application, here: http://myjob.io/nd6qd You can find all jobs, and information about working at St. Jude, here: https://www.stjude.org/jobs/hospital.html Please feel free to contact me directly off list if you have any questions. I?ll be at SC this year, and also presenting at the user group meeting, and hope to see you there. Thanks, Will ________________________________ Email Disclaimer: www.stjude.org/emaildisclaimer Consultation Disclaimer: www.stjude.org/consultationdisclaimer -------------- next part -------------- An HTML attachment was scrubbed... URL: From scale at us.ibm.com Tue Nov 12 08:16:29 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 12 Nov 2019 16:16:29 +0800 Subject: [gpfsug-discuss] Force removal of problematic inode? In-Reply-To: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> References: <892217F9-D15C-4051-AEC8-4305D43399D6@nuance.com> Message-ID: You can try if restarting GPFS daemon would help. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Oesterlin, Robert" To: gpfsug main discussion list Date: 11/11/2019 09:28 PM Subject: [EXTERNAL] [gpfsug-discuss] Force removal of problematic inode? Sent by: gpfsug-discuss-bounces at spectrumscale.org Any ideas on how I could get out of this, short of a full mmfsck? Directory paths shortened to obfuscate data. [root at nrg5-gpfs01 remove]# tsfindinode -i 11198927 /gpfs/lm gpfs_iopen(/gpfs/lm/.../build_respell_map)(184769551:0xD4F8): Device or resource busy ^C ls -li /gpfs/lm/?/ ls: cannot access /gpfs/lm/.../build_respell_map: Device or resource busy total 0 ? d????????? ? ? ? ? ? build_respell_map [root at nrg5-gpfs01 remove]# rm -rf /gpfs/lm/.../* rm: cannot remove ?/gpfs/lm/.../build_respell_map?: Device or resource busy Bob Oesterlin Sr Principal Storage Engineer, Nuance _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=Pid3Qt9isGIuPoObkvy-RLwrhUmEM-mNU4YAfNWpDRo&s=TMRis9DxrIhg7D9xwf4IYi0tNH1RrWqmeSL1cDD1hi4&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From scale at us.ibm.com Tue Nov 12 08:18:24 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Tue, 12 Nov 2019 16:18:24 +0800 Subject: [gpfsug-discuss] orphaned deleted fileset In-Reply-To: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> References: <4c5cc66c20204d2eafe259c8bfa9913b@id.ethz.ch> Message-ID: run mmfsck or try to delete another fileset to see if that could trigger cleanup. Thanks. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Zimmermann Alexander (ID SD)" To: "gpfsug-discuss at spectrumscale.org" Date: 11/11/2019 10:16 PM Subject: [EXTERNAL] [gpfsug-discuss] orphaned deleted fileset Sent by: gpfsug-discuss-bounces at spectrumscale.org Hi, we?ve an orphaned deleted fileset in one of our filesystems: [azi4ea at nas22ems01 ~]$ mmlsfileset /dev/fs2201 --deleted Filesets in file system 'fs2201': Name Status Path (chab_services_admin_s1) Deleted /fs2201/.snapshots/@GMT-2019.10.21-10.45.55/chab_services_admin_s1 (biol_bc_azzalin_1) Deleted -- [azi4ea at nas22ems01 ~]$ biol_bc_azzalin_1 is now in that state for roughly 1 week, initially it was in the state Deleting until I ran (another?) mmdelfileset command on it. The order to delete it in our automation tool was completed in July so it?s hard find out what went wrong back then. Will it get cleaned up on its own and I?m only too impatient or do I need to intervene to get rid of it completely? Many thanks in advance for your aid! Kind regards, Alexander Zimmermann Alexander Zimmermann ETH Z?rich ID Systemdienste - Speicher Weinbergstrasse 11, WEC C16 8092 Z?rich Tel.: +41 44 632 72 39 E-Mail: alexander.zimmermann at id.ethz.ch www.id.ethz.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=dy9kUmE1i9dh2VjQBu8Cqis2Us0lfMxVc_OzA2Jwhfk&s=eazf8OnQPmH23k5Z0VwYOGDLa5-LL88CcvdUqctGMIY&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From S.J.Thompson at bham.ac.uk Tue Nov 12 14:49:07 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Tue, 12 Nov 2019 14:49:07 +0000 Subject: [gpfsug-discuss] SC19 SSUG Lightning talks Message-ID: <5C96F4C4-13B8-41A7-8834-9D32928AE738@bham.ac.uk> Hi All, It?s less than a week until the Spectrum Scale user group at SC19 in Denver! As part of the agenda, we have included a lightning talk slot and I?m looking for people to volunteer to do a zero slide talk on something related to Scale. Those at the UK event may remember we ran something similar on ?my favourite/worst feature of Spectrum Scale?. I?m happy to use the same topic, or for people to do a 3-5 minute lightning slot on Scale and why they use it ? or some other Scale topic entirely! If you are interested, I?ll be around on Sunday at the SSUG and you can let me know then/volunteer in the slot. If no-one volunteers, we?ll have a very quiet 30 mins in the user group session! Please have a think on something you might want to say on the day! Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From Rafael.Cezario at ibm.com Wed Nov 13 17:24:34 2019 From: Rafael.Cezario at ibm.com (Rafael Cezario) Date: Wed, 13 Nov 2019 14:24:34 -0300 Subject: [gpfsug-discuss] Online migration Message-ID: Hello, I have a Spectrum Scale cluster with 10 nodes on the version 4.2. The cluster have maxblocksize=DEFAULT. I will migrate all nodes to 5.0.3 (online) none downtime. But the maxblocksize is fixed on the mmlsconfig after the migration (the man have this information) maxblocksize 1M man mmchconfig Note: When you migrate a cluster from an earlier version to 5.0.0 or later, the value of maxblocksize stays the same. However, if maxblocksize was set to DEFAULT in the earlier version of the cluster, then migrating it to 5.0.0 or later sets it explicitly to 1 MiB, which was the default value in earlier versions. To change maxblocksize to the default value after migrating to 5.0.0 or later, set maxblocksize=DEFAULT (4 MiB). How Can I perform non-stop migration and after migration create a new filesystem with blocksize = 4MiB? My question is about finishing the migration and create a new filesystem with block size set to 4 MiB, its won't be possible until the maxblocksize changes. In a environment where all migration will be done without stopping the cluster, my question was if there was any alternative This behavior forces the entire cluster to stop. Regards, Rafael Cez?rio Dam?sio Cunha IT Specialist - PowerVM/AIX/Spectrum Scale Technology Support Services Global Technology Services Mobile: 55-6198122-4435 E-mail: rafael.cezario at ibm.com Scn Quadra 4, Bloco B, n.100 Brasilia, DF 70714-900 Brazil -------------- next part -------------- An HTML attachment was scrubbed... URL: From knop at us.ibm.com Wed Nov 13 19:05:00 2019 From: knop at us.ibm.com (Felipe Knop) Date: Wed, 13 Nov 2019 19:05:00 +0000 Subject: [gpfsug-discuss] Online migration In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From andi at christiansen.xxx Wed Nov 13 21:33:57 2019 From: andi at christiansen.xxx (Andi Christiansen) Date: Wed, 13 Nov 2019 22:33:57 +0100 (CET) Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. Message-ID: <384482306.75861.1573680837216@privateemail.com> An HTML attachment was scrubbed... URL: From bbanister at jumptrading.com Wed Nov 13 22:05:08 2019 From: bbanister at jumptrading.com (Bryan Banister) Date: Wed, 13 Nov 2019 22:05:08 +0000 Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. In-Reply-To: <384482306.75861.1573680837216@privateemail.com> References: <384482306.75861.1573680837216@privateemail.com> Message-ID: Hello, This is simply due to the fact that your ?test? is doing parallel appended writes to the same file. With GPFS, there is one node at a time that can write to roughly the same position (aka byte range) in a file. This is handled by something called a write_lock token that has to be revoked on the node that currently owns the token, then given to one of the other nodes attempting to perform the write. This overhead in lock token management is done for almost every write that is happening in this ?test?. This is not something that you would ever really want an application to do anyways. GPFS offers byte-range locking to allow for very fast, non-blocking I/O to the same file from multiple nodes in parallel. So if you change your ?test? so that each node seeks to a different part of the single file and that each node writes into the file in a way that there is no overlap in the byte range of your total writes from the node, then GPFS will give a byte-range write_lock token to each node. Since no node will attempt to write to the same part of the file as another node (which is what you?d want anyways in most cases) then there will be no write_lock token revoking and no blocking of your I/O to the single file. Hope that helps, -Bryan From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Andi Christiansen Sent: Wednesday, November 13, 2019 3:34 PM To: gpfsug main discussion list Subject: [gpfsug-discuss] Writing to the same file from multiple clients gives slow performance compared to beegfs. [EXTERNAL EMAIL] Hi all, I am in the process of replacing a beegfs cluster with a spectrum scale cluster and some of our initial tests have returned poor performance when writing from multiple client nodes to the same file. If we setup a client to write into a file it takes less than 8 seconds to complete the write on Flash and about the same on NLSAS storage. But if we get 3 more nodes to do the exact same write the cluster seems to slow down and completes all writes in about 1.5 minute. We are running 5.0.4-0 on 4 Lenovo SR630 servers with a V7000 control enclosure with flash drives and a 92F drawer with NLSAS drives behind the nodes attach through SAN. Is there something I am missing since the writes are so slow compared to beegfs? Beegfs completes the write from all clients within 9 second when run from 4 clients doing the same write to the same file GPFS takes 1.5 min to do the same. Tests run: time (for i in `seq 5000`;do echo "This is $(hostname) writing line number" $i >> "/gpfs_T0/test/benchmark1.txt";done) ########## this is run from 4 gpfs client nodes at the same time. Result for scale: real 1m43.995s user 0m0.821s sys 0m3.545s Result for beegfs: real 0m6.507s user 0m0.651s sys 0m2.534s if we run writes from each client node to separate files, performance is way better with GPFS than beegfs but not when the writes are done parallel. If anyone have an idea I would be glad to hear it ? Best Regards Andi Christiansen -------------- next part -------------- An HTML attachment was scrubbed... URL: From tshukla at NYGENOME.ORG Wed Nov 20 15:36:16 2019 From: tshukla at NYGENOME.ORG (Tushit Shukla) Date: Wed, 20 Nov 2019 15:36:16 +0000 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 81, Issue 43 In-Reply-To: References: Message-ID: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> Lohit, Did you get any progress on file system performance issue with large block size ? Apparently we are also going through the same issue after we migrated to 16M file system (from 4MB filesystem). Only thing which gives some relief is when we increase pagepool to 32G. We also tried playing with prefetchAggressivenessRead parameter by reducing it to 1 but not sure if it is doing anything. thanks ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org Sent: Monday, October 22, 2018 12:18:58 PM To: gpfsug-discuss at spectrumscale.org Subject: gpfsug-discuss Digest, Vol 81, Issue 43 Send gpfsug-discuss mailing list submissions to gpfsug-discuss at spectrumscale.org To subscribe or unsubscribe via the World Wide Web, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= or, via email, send a message with subject or body 'help' to gpfsug-discuss-request at spectrumscale.org You can reach the person managing the list at gpfsug-discuss-owner at spectrumscale.org When replying, please edit your Subject line so it is more specific than "Re: Contents of gpfsug-discuss digest..." Today's Topics: 1. Re: GPFS, Pagepool and Block size -> Perfomance reduces with larger block size (Sven Oehme) ---------------------------------------------------------------------- Message: 1 Date: Mon, 22 Oct 2018 09:18:43 -0700 From: Sven Oehme To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] GPFS, Pagepool and Block size -> Perfomance reduces with larger block size Message-ID: Content-Type: text/plain; charset="utf-8" oops, somehow that slipped my inbox, i only saw that reply right now. its really hard to see from a trace snipped if the lock is the issue as the lower level locks don't show up in default traces. without having access to source code and a detailed trace you won't make much progress here. sven On Thu, Sep 27, 2018 at 12:31 PM wrote: > Thank you Sven, > > Turning of prefetching did not improve the performance, but it did degrade > a bit. > > I have made the prefetching default and took trace dump, for tracectl with > trace=io. Let me know if you want me to paste/attach it here. > > May i know, how could i confirm if the below is true? > > 1. this could be serialization around buffer locks. as larger your >>> blocksize gets as larger is the amount of data one of this pagepool buffers >>> will maintain, if there is a lot of concurrency on smaller amount of data >>> more threads potentially compete for the same buffer lock to copy stuff in >>> and out of a particular buffer, hence things go slower compared to the same >>> amount of data spread across more buffers, each of smaller size. >>> >>> > Will the above trace help in understanding if it is a serialization issue? > > I had been discussing the same with GPFS support for past few months, and > it seems to be that most of the time is being spent at cxiUXfer. They could > not understand on why it is taking spending so much of time in cxiUXfer. I > was seeing the same from perf top, and pagefaults. > > Below is snippet from what the support had said : > > ???????????????????????????? > > I searched all of the gpfsRead from trace and sort them by spending-time. > Except 2 reads which need fetch data from nsd server, the slowest read is > in the thread 72170. It took 112470.362 us. > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum: 72165 6.860911319 > rdwr 141857.076 us + NSDIO > > trcrpt.2018-08-06_12.26.28.39794.lt15.trsum: 72170 1.483947593 > rdwr 112470.362 us + cxiUXfer > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum: 72165 6.949042593 > rdwr 88126.278 us + NSDIO > > trcrpt.2018-08-06_12.27.03.47706.lt15.trsum: 72156 2.919334474 > rdwr 81057.657 us + cxiUXfer > > trcrpt.2018-08-06_12.23.30.72745.lt15.trsum: 72154 1.167484466 > rdwr 76033.488 us + cxiUXfer > > trcrpt.2018-08-06_12.24.06.7508.lt15.trsum: 72187 0.685237501 > rdwr 70772.326 us + cxiUXfer > > trcrpt.2018-08-06_12.25.17.23989.lt15.trsum: 72193 4.757996530 > rdwr 70447.838 us + cxiUXfer > > > I check each of the slow IO as above, and find they all spend much time in > the function cxiUXfer. This function is used to copy data from kernel > buffer to user buffer. I am not sure why it took so much time. This should > be related to the pagefaults and pgfree you observed. Below is the trace > data for thread 72170. > > > 1.371477231 72170 TRACE_VNODE: gpfs_f_rdwr enter: fP > 0xFFFF882541649400 f_flags 0x8000 flags 0x8001 op 0 iovec > 0xFFFF881F2AFB3E70 count 1 offset 0x168F30D dentry 0xFFFF887C0CC298C0 > private 0xFFFF883F607175C0 iP 0xFFFF8823AA3CBFC0 name '410513.svs' > > .... > > 1.371483547 72170 TRACE_KSVFS: cachedReadFast exit: > uio_resid 16777216 code 1 err 11 > > .... > > 1.371498780 72170 TRACE_KSVFS: kSFSReadFast: oiP > 0xFFFFC90060B46740 offset 0x168F30D dataBufP FFFFC9003645A5A8 nDesc 64 buf > 200043C0000 valid words 64 dirty words 0 blkOff 0 > > 1.371499035 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate begin ul 0xFFFFC900333F1A40 holdCount 0 > ioType 0x2 inProg 0x15 > > 1.371500157 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate ul 0xFFFFC900333F1A40 holdCount 0 ioType 0x2 > inProg 0x16 err 0 > > 1.371500606 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 toIOBuf 0 offset 6877965 len > 9899251 > > 1.371500793 72170 TRACE_KSVFS: cxiUXfer: ndesc 0 skip > dataAddrP 0x200043C0000 currOffset 0 currLen 262144 bufOffset 6877965 > > .... > > 1.371505949 72170 TRACE_KSVFS: cxiUXfer: ndesc 25 skip > dataAddrP 0x2001AF80000 currOffset 6553600 currLen 262144 bufOffset 6877965 > > 1.371506236 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > currOffset 6815744 tmpLen 262144 dataAddrP 0x2001AFCF30D currLen 199923 > pageOffset 781 pageLen 3315 plP 0xFFFF887F7B90D600 > > 1.373649823 72170 TRACE_KSVFS: cxiUXfer: nDesc 27 > currOffset 7077888 tmpLen 262144 dataAddrP 0x20027400000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.375158799 72170 TRACE_KSVFS: cxiUXfer: nDesc 28 > currOffset 7340032 tmpLen 262144 dataAddrP 0x20027440000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.376661566 72170 TRACE_KSVFS: cxiUXfer: nDesc 29 > currOffset 7602176 tmpLen 262144 dataAddrP 0x2002C180000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.377892653 72170 TRACE_KSVFS: cxiUXfer: nDesc 30 > currOffset 7864320 tmpLen 262144 dataAddrP 0x2002C1C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > .... > > 1.471389843 72170 TRACE_KSVFS: cxiUXfer: nDesc 62 > currOffset 16252928 tmpLen 262144 dataAddrP 0x2001D2C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.471845629 72170 TRACE_KSVFS: cxiUXfer: nDesc 63 > currOffset 16515072 tmpLen 262144 dataAddrP 0x2003EC80000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > 1.472417149 72170 TRACE_KSVFS: cxiDetachIOBuffer: > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 > > 1.472417775 72170 TRACE_LOCK: unlock_vfs: type Data, > key 0000000000000004:000000001B1F24BF:0000000000000001 lock_mode have ro > token xw lock_state old [ ro:27 ] new [ ro:26 ] holdCount now 27 > > 1.472418427 72170 TRACE_LOCK: hash tab lookup vfs: > found cP 0xFFFFC9005FC0CDE0 holdCount now 14 > > 1.472418592 72170 TRACE_LOCK: lock_vfs: type Data key > 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode want ro status > valid token xw/xw lock_state [ ro:12 ] flags 0x0 holdCount 14 > > 1.472419842 72170 TRACE_KSVFS: kSFSReadFast: oiP > 0xFFFFC90060B46740 offset 0x2000000 dataBufP FFFFC9003643C908 nDesc 64 buf > 38033480000 valid words 64 dirty words 0 blkOff 0 > > 1.472420029 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate begin ul 0xFFFFC9005FC0CF98 holdCount 0 > ioType 0x2 inProg 0xC > > 1.472420187 72170 TRACE_LOG: > UpdateLogger::beginDataUpdate ul 0xFFFFC9005FC0CF98 holdCount 0 ioType 0x2 > inProg 0xD err 0 > > 1.472420652 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 toIOBuf 0 offset 0 len 6877965 > > 1.472420936 72170 TRACE_KSVFS: cxiUXfer: nDesc 0 > currOffset 0 tmpLen 262144 dataAddrP 0x38033480000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.472824790 72170 TRACE_KSVFS: cxiUXfer: nDesc 1 > currOffset 262144 tmpLen 262144 dataAddrP 0x380334C0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.473243905 72170 TRACE_KSVFS: cxiUXfer: nDesc 2 > currOffset 524288 tmpLen 262144 dataAddrP 0x38024280000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > .... > > 1.482949347 72170 TRACE_KSVFS: cxiUXfer: nDesc 24 > currOffset 6291456 tmpLen 262144 dataAddrP 0x38025E80000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483354265 72170 TRACE_KSVFS: cxiUXfer: nDesc 25 > currOffset 6553600 tmpLen 262144 dataAddrP 0x38025EC0000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483766631 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > currOffset 6815744 tmpLen 262144 dataAddrP 0x38003B00000 currLen 262144 > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > 1.483943894 72170 TRACE_KSVFS: cxiDetachIOBuffer: > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 > > 1.483944339 72170 TRACE_LOCK: unlock_vfs: type Data, > key 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode have ro > token xw lock_state old [ ro:14 ] new [ ro:13 ] holdCount now 14 > > 1.483944683 72170 TRACE_BRL: brUnlockM: ofP > 0xFFFFC90069346B68 inode 455025855 snap 0 handle 0xFFFFC9003637D020 range > 0x168F30D-0x268F30C mode ro > > 1.483944985 72170 TRACE_KSVFS: kSFSReadFast exit: > uio_resid 0 err 0 > > 1.483945264 72170 TRACE_LOCK: unlock_vfs_m: type > Inode, key 305F105B9701E60A:000000001B1F24BF:0000000000000000 lock_mode > have ro status valid token rs lock_state old [ ro:25 ] new [ ro:24 ] > > 1.483945423 72170 TRACE_LOCK: unlock_vfs_m: cP > 0xFFFFC90069346B68 holdCount 25 > > 1.483945624 72170 TRACE_VNODE: gpfsRead exit: fast err > 0 > > 1.483946831 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > 0xFFFFC90035E52F78 NotQuiesced vfsOp 2 > > 1.483946975 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > 0xFFFFC90035E52F78 vfsOp 2 users 1-1 > > 1.483947116 72170 TRACE_KSVFS: ReleaseDaemonSegAndSG: > sli 38 count 2 needCleanup 0 > > 1.483947593 72170 TRACE_VNODE: gpfs_f_rdwr exit: fP > 0xFFFF882541649400 total_len 16777216 uio_resid 0 offset 0x268F30D rc 0 > > > ??????????????????????????????????????????? > > > > Regards, > Lohit > > On Sep 19, 2018, 3:11 PM -0400, Sven Oehme , wrote: > > the document primarily explains all performance specific knobs. general > advice would be to longer set anything beside workerthreads, pagepool and > filecache on 5.X systems as most other settings are no longer relevant > (thats a client side statement) . thats is true until you hit strange > workloads , which is why all the knobs are still there :-) > > sven > > > On Wed, Sep 19, 2018 at 11:17 AM wrote: > >> Thanks Sven. >> I will disable it completely and see how it behaves. >> >> Is this the presentation? >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2014_UG10-5FGPFS-5FPerformance-5FSession-5Fv10.pdf&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=-Y1ncRDmgOCfWDQaXj-OcbiM5gcs0LcllAShfNapHtI&e= >> >> I guess i read it, but it did not strike me at this situation. I will try >> to read it again and see if i could make use of it. >> >> Regards, >> Lohit >> >> On Sep 19, 2018, 2:12 PM -0400, Sven Oehme , wrote: >> >> seem like you never read my performance presentation from a few years ago >> ;-) >> >> you can control this on a per node basis , either for all i/o : >> >> prefetchAggressiveness = X >> >> or individual for reads or writes : >> >> prefetchAggressivenessRead = X >> prefetchAggressivenessWrite = X >> >> for a start i would turn it off completely via : >> >> mmchconfig prefetchAggressiveness=0 -I -N nodename >> >> that will turn it off only for that node and only until you restart the >> node. >> then see what happens >> >> sven >> >> >> On Wed, Sep 19, 2018 at 11:07 AM wrote: >> >>> Thank you Sven. >>> >>> I mostly think it could be 1. or some other issue. >>> I don?t think it could be 2. , because i can replicate this issue no >>> matter what is the size of the dataset. It happens for few files that could >>> easily fit in the page pool too. >>> >>> I do see a lot more page faults for 16M compared to 1M, so it could be >>> related to many threads trying to compete for the same buffer space. >>> >>> I will try to take the trace with trace=io option and see if can find >>> something. >>> >>> How do i turn of prefetching? Can i turn it off for a single >>> node/client? >>> >>> Regards, >>> Lohit >>> >>> On Sep 18, 2018, 5:23 PM -0400, Sven Oehme , wrote: >>> >>> Hi, >>> >>> taking a trace would tell for sure, but i suspect what you might be >>> hitting one or even multiple issues which have similar negative performance >>> impacts but different root causes. >>> >>> 1. this could be serialization around buffer locks. as larger your >>> blocksize gets as larger is the amount of data one of this pagepool buffers >>> will maintain, if there is a lot of concurrency on smaller amount of data >>> more threads potentially compete for the same buffer lock to copy stuff in >>> and out of a particular buffer, hence things go slower compared to the same >>> amount of data spread across more buffers, each of smaller size. >>> >>> 2. your data set is small'ish, lets say a couple of time bigger than the >>> pagepool and you random access it with multiple threads. what will happen >>> is that because it doesn't fit into the cache it will be read from the >>> backend. if multiple threads hit the same 16 mb block at once with multiple >>> 4k random reads, it will read the whole 16mb block because it thinks it >>> will benefit from it later on out of cache, but because it fully random the >>> same happens with the next block and the next and so on and before you get >>> back to this block it was pushed out of the cache because of lack of enough >>> pagepool. >>> >>> i could think of multiple other scenarios , which is why its so hard to >>> accurately benchmark an application because you will design a benchmark to >>> test an application, but it actually almost always behaves different then >>> you think it does :-) >>> >>> so best is to run the real application and see under which configuration >>> it works best. >>> >>> you could also take a trace with trace=io and then look at >>> >>> TRACE_VNOP: READ: >>> TRACE_VNOP: WRITE: >>> >>> and compare them to >>> >>> TRACE_IO: QIO: read >>> TRACE_IO: QIO: write >>> >>> and see if the numbers summed up for both are somewhat equal. if >>> TRACE_VNOP is significant smaller than TRACE_IO you most likely do more i/o >>> than you should and turning prefetching off might actually make things >>> faster . >>> >>> keep in mind i am no longer working for IBM so all i say might be >>> obsolete by now, i no longer have access to the one and only truth aka the >>> source code ... but if i am wrong i am sure somebody will point this out >>> soon ;-) >>> >>> sven >>> >>> >>> >>> >>> On Tue, Sep 18, 2018 at 10:31 AM wrote: >>> >>>> Hello All, >>>> >>>> This is a continuation to the previous discussion that i had with Sven. >>>> However against what i had mentioned previously - i realize that this >>>> is ?not? related to mmap, and i see it when doing random freads. >>>> >>>> I see that block-size of the filesystem matters when reading from Page >>>> pool. >>>> I see a major difference in performance when compared 1M to 16M, when >>>> doing lot of random small freads with all of the data in pagepool. >>>> >>>> Performance for 1M is a magnitude ?more? than the performance that i >>>> see for 16M. >>>> >>>> The GPFS that we have currently is : >>>> Version : 5.0.1-0.5 >>>> Filesystem version: 19.01 (5.0.1.0) >>>> Block-size : 16M >>>> >>>> I had made the filesystem block-size to be 16M, thinking that i would >>>> get the most performance for both random/sequential reads from 16M than the >>>> smaller block-sizes. >>>> With GPFS 5.0, i made use the 1024 sub-blocks instead of 32 and thus >>>> not loose lot of storage space even with 16M. >>>> I had run few benchmarks and i did see that 16M was performing better >>>> ?when hitting storage/disks? with respect to bandwidth for >>>> random/sequential on small/large reads. >>>> >>>> However, with this particular workload - where it freads a chunk of >>>> data randomly from hundreds of files -> I see that the number of >>>> page-faults increase with block-size and actually reduce the performance. >>>> 1M performs a lot better than 16M, and may be i will get better >>>> performance with less than 1M. >>>> It gives the best performance when reading from local disk, with 4K >>>> block size filesystem. >>>> >>>> What i mean by performance when it comes to this workload - is not the >>>> bandwidth but the amount of time that it takes to do each iteration/read >>>> batch of data. >>>> >>>> I figure what is happening is: >>>> fread is trying to read a full block size of 16M - which is good in a >>>> way, when it hits the hard disk. >>>> But the application could be using just a small part of that 16M. Thus >>>> when randomly reading(freads) lot of data of 16M chunk size - it is page >>>> faulting a lot more and causing the performance to drop . >>>> I could try to make the application do read instead of freads, but i >>>> fear that could be bad too since it might be hitting the disk with a very >>>> small block size and that is not good. >>>> >>>> With the way i see things now - >>>> I believe it could be best if the application does random reads of >>>> 4k/1M from pagepool but some how does 16M from rotating disks. >>>> >>>> I don?t see any way of doing the above other than following a different >>>> approach where i create a filesystem with a smaller block size ( 1M or less >>>> than 1M ), on SSDs as a tier. >>>> >>>> May i please ask for advise, if what i am understanding/seeing is right >>>> and the best solution possible for the above scenario. >>>> >>>> Regards, >>>> Lohit >>>> >>>> On Apr 11, 2018, 10:36 AM -0400, Lohit Valleru , >>>> wrote: >>>> >>>> Hey Sven, >>>> >>>> This is regarding mmap issues and GPFS. >>>> We had discussed previously of experimenting with GPFS 5. >>>> >>>> I now have upgraded all of compute nodes and NSD nodes to GPFS 5.0.0.2 >>>> >>>> I am yet to experiment with mmap performance, but before that - I am >>>> seeing weird hangs with GPFS 5 and I think it could be related to mmap. >>>> >>>> Have you seen GPFS ever hang on this syscall? >>>> [Tue Apr 10 04:20:13 2018] [] >>>> _ZN10gpfsNode_t8mmapLockEiiPKj+0xb5/0x140 [mmfs26] >>>> >>>> I see the above ,when kernel hangs and throws out a series of trace >>>> calls. >>>> >>>> I somehow think the above trace is related to processes hanging on GPFS >>>> forever. There are no errors in GPFS however. >>>> >>>> Also, I think the above happens only when the mmap threads go above a >>>> particular number. >>>> >>>> We had faced a similar issue in 4.2.3 and it was resolved in a patch to >>>> 4.2.3.2 . At that time , the issue happened when mmap threads go more than >>>> worker1threads. According to the ticket - it was a mmap race condition that >>>> GPFS was not handling well. >>>> >>>> I am not sure if this issue is a repeat and I am yet to isolate the >>>> incident and test with increasing number of mmap threads. >>>> >>>> I am not 100 percent sure if this is related to mmap yet but just >>>> wanted to ask you if you have seen anything like above. >>>> >>>> Thanks, >>>> >>>> Lohit >>>> >>>> On Feb 22, 2018, 3:59 PM -0500, Sven Oehme , wrote: >>>> >>>> Hi Lohit, >>>> >>>> i am working with ray on a mmap performance improvement right now, >>>> which most likely has the same root cause as yours , see --> >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_2018-2DJanuary_004411.html&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=AUEL847F_a-j6Y7t1fDMj4j33vLqvI6XrrNCVS5pUyA&e= >>>> the thread above is silent after a couple of back and rorth, but ray >>>> and i have active communication in the background and will repost as soon >>>> as there is something new to share. >>>> i am happy to look at this issue after we finish with ray's workload if >>>> there is something missing, but first let's finish his, get you try the >>>> same fix and see if there is something missing. >>>> >>>> btw. if people would share their use of MMAP , what applications they >>>> use (home grown, just use lmdb which uses mmap under the cover, etc) please >>>> let me know so i get a better picture on how wide the usage is with GPFS. i >>>> know a lot of the ML/DL workloads are using it, but i would like to know >>>> what else is out there i might not think about. feel free to drop me a >>>> personal note, i might not reply to it right away, but eventually. >>>> >>>> thx. sven >>>> >>>> >>>> On Thu, Feb 22, 2018 at 12:33 PM wrote: >>>> >>>>> Hi all, >>>>> >>>>> I wanted to know, how does mmap interact with GPFS pagepool with >>>>> respect to filesystem block-size? >>>>> Does the efficiency depend on the mmap read size and the block-size of >>>>> the filesystem even if all the data is cached in pagepool? >>>>> >>>>> GPFS 4.2.3.2 and CentOS7. >>>>> >>>>> Here is what i observed: >>>>> >>>>> I was testing a user script that uses mmap to read from 100M to 500MB >>>>> files. >>>>> >>>>> The above files are stored on 3 different filesystems. >>>>> >>>>> Compute nodes - 10G pagepool and 5G seqdiscardthreshold. >>>>> >>>>> 1. 4M block size GPFS filesystem, with separate metadata and data. >>>>> Data on Near line and metadata on SSDs >>>>> 2. 1M block size GPFS filesystem as a AFM cache cluster, "with all the >>>>> required files fully cached" from the above GPFS cluster as home. Data and >>>>> Metadata together on SSDs >>>>> 3. 16M block size GPFS filesystem, with separate metadata and data. >>>>> Data on Near line and metadata on SSDs >>>>> >>>>> When i run the script first time for ?each" filesystem: >>>>> I see that GPFS reads from the files, and caches into the pagepool as >>>>> it reads, from mmdiag -- iohist >>>>> >>>>> When i run the second time, i see that there are no IO requests from >>>>> the compute node to GPFS NSD servers, which is expected since all the data >>>>> from the 3 filesystems is cached. >>>>> >>>>> However - the time taken for the script to run for the files in the 3 >>>>> different filesystems is different - although i know that they are just >>>>> "mmapping"/reading from pagepool/cache and not from disk. >>>>> >>>>> Here is the difference in time, for IO just from pagepool: >>>>> >>>>> 20s 4M block size >>>>> 15s 1M block size >>>>> 40S 16M block size. >>>>> >>>>> Why do i see a difference when trying to mmap reads from different >>>>> block-size filesystems, although i see that the IO requests are not hitting >>>>> disks and just the pagepool? >>>>> >>>>> I am willing to share the strace output and mmdiag outputs if needed. >>>>> >>>>> Thanks, >>>>> Lohit >>>>> >>>>> _______________________________________________ >>>>> gpfsug-discuss mailing list >>>>> gpfsug-discuss at spectrumscale.org >>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>>> _______________________________________________ >>>> gpfsug-discuss mailing list >>>> gpfsug-discuss at spectrumscale.org >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>> >>> _______________________________________________ >>> gpfsug-discuss mailing list >>> gpfsug-discuss at spectrumscale.org >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >>> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= >> > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= End of gpfsug-discuss Digest, Vol 81, Issue 43 ********************************************** ________________________________ This message is for the recipient's use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. -------------- next part -------------- An HTML attachment was scrubbed... URL: From valleru at cbio.mskcc.org Wed Nov 20 16:46:03 2019 From: valleru at cbio.mskcc.org (valleru at cbio.mskcc.org) Date: Wed, 20 Nov 2019 11:46:03 -0500 Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 81, Issue 43 In-Reply-To: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> References: <8363e36f15074b74933ba554035cc443@NYGENOME.ORG> Message-ID: <2ff249fe-15d5-4d48-88fa-12ba40d1a0f8@Spark> Hey Tushit, For us, it finally came down to the kind of applications that are doing IO. For some applications, the 16M filesystem was very bad in performance, while for some it seems to do well. Yes, I do observe a performance difference with higher pagepool but the amount of pagepool also happens to depend on the working dataset of the applications. For us - Most of the compute nodes run a variety of applications with varying datasets. Thus modifying pagepool to a large value for all nodes did not seem to be practical/enough for us. I have not made prefetchaggressivenessread/large pagepool for every node yet. I am setting it only if it is making a big difference and only for dedicated compute nodes that run only specific set of applications. I believe the issue that I was facing with respect to large block size might be because of few things such as: 1. The amount of buffers/cache that can be held in a particular amount of pagepool. 2. If the application needs to read from the same buffers multiple times or from different buffers thus deciding the need for buffer locking issue as Sven mentioned or may be not enough buffers because of large block size and not enough pagepool memory, thus needing to page in/page out and read over the network to the storage. 3. The latency to read the full block size/16M from the storage and if the application is ok with this latency. I see that most of the GPU deep learning applications are not happy with large block size latency/IOPS/reads per second. 4. Most of the scientific applications including some libraries are designed to do buffered reads (full block size) from local disk and local disk mostly has ext4 4K block size so it should not affect much when done from local disk. But with GPFS 16M filesystem - they will read 16M from the GPFS filesystem/over the network ( though they don?t need all of the data in each 16M read). Thus with prefetch, and depending on IO pattern - GPFS can fetch lots of 16M reads unnecessarily needing for the OS to discard the rest of the data once it receives. In short - I had seen considerable difference when I had modified the applications to do unbuffered reads instead of buffered reads from a large block size filesystem. 5. The large block size reads also affected the Arista 100G network switch introducing lot of network discards, and the need for a higher buffer switch or enable ECN. 6. If the application does large sequential IO reads/large random IO reads or small random IO reads from many different files. The latter applications seems to be happy with smaller block size/larger pagepool/more maxfilestocache/network switch with lesser latency. While the former applications seem to do ok with large block size. Thus we have decided to debug/strace applications one a time, and see which applications are being affected by large block size and test them with a smaller block size on a SSD/Flash backed storage. Just yesterday, I was debugging one GPU deep learning application that does lot of small random IO from thousands of different files/latency sensitive and it performed best with small block size/large pagepool and more max files to cache/prefetchaggressivenessread to zero. The reason we had moved to 16M filesystem was only because of the reason that the rotating disk storage was most happy and gives maximum throughput when on larger block size ( ignoring the latency that this might introduce for each read). However we realized that not every application would be suited for a large block size, because of the above mentioned reasons. I had also seen that those applications are usually happy from local disk, because of the smaller block size that local disk offers and the dynamic filebuffer/cache that linux offers ( unlike pagepool which is static and pinned ). The filebuffer/cache has its advantages and disadvantages because of the varying performance depending on the free memory available. 4M block size might work better than 16M for such applications, but that will also reduce the total throughput that we get from the rotating disk storage - and not give much benefit with respect to latency/IOPS. So it all depends on the applications/IO pattern, and as of now - we are in process of gathering which applications are suited for large vs smaller block size. All the above is ignoring the metadata overhead/performance. Sometimes depending on the applications - metadata performance can be a bottleneck too. Regards, Lohit On Nov 20, 2019, 10:43 AM -0500, Tushit Shukla , wrote: > Lohit, > Did you get any progress on file system?performance issue with large block size ? > Apparently we are also going through the same issue after we migrated to 16M file system (from 4MB filesystem). Only thing which gives some relief is when we increase pagepool to 32G. We also tried playing with prefetchAggressivenessRead parameter by reducing it to 1 but not sure if it is doing anything. > > thanks > From: gpfsug-discuss-bounces at spectrumscale.org on behalf of gpfsug-discuss-request at spectrumscale.org > Sent: Monday, October 22, 2018 12:18:58 PM > To: gpfsug-discuss at spectrumscale.org > Subject: gpfsug-discuss Digest, Vol 81, Issue 43 > > Send gpfsug-discuss mailing list submissions to > ??????? gpfsug-discuss at spectrumscale.org > > To subscribe or unsubscribe via the World Wide Web, visit > ??????? https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > or, via email, send a message with subject or body 'help' to > ??????? gpfsug-discuss-request at spectrumscale.org > > You can reach the person managing the list at > ??????? gpfsug-discuss-owner at spectrumscale.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of gpfsug-discuss digest..." > > > Today's Topics: > > ?? 1. Re: GPFS, Pagepool and Block size -> Perfomance reduces with > ????? larger block size (Sven Oehme) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 22 Oct 2018 09:18:43 -0700 > From: Sven Oehme > To: gpfsug main discussion list > Subject: Re: [gpfsug-discuss] GPFS, Pagepool and Block size -> > ??????? Perfomance reduces with larger block size > Message-ID: > ??????? > Content-Type: text/plain; charset="utf-8" > > oops, somehow that slipped my inbox, i only saw that reply right now. > > its really hard to see from a trace snipped if the lock is the issue as the > lower level locks don't show up in default traces. without having access to > source code and a detailed trace you won't make much progress here. > > sven > > > > > > On Thu, Sep 27, 2018 at 12:31 PM wrote: > > > Thank you Sven, > > > > Turning of prefetching did not improve the performance, but it did degrade > > a bit. > > > > I have made the prefetching default and took trace dump, for tracectl with > > trace=io. Let me know if you want me to paste/attach it here. > > > > May i know, how could i confirm if the below is true? > > > > 1. this could be serialization around buffer locks. as larger your > >>> blocksize gets as larger is the amount of data one of this pagepool buffers > >>> will maintain, if there is a lot of concurrency on smaller amount of data > >>> more threads potentially compete for the same buffer lock to copy stuff in > >>> and out of a particular buffer, hence things go slower compared to the same > >>> amount of data spread across more buffers, each of smaller size. > >>> > >>> > > Will the above trace help in understanding if it is a serialization issue? > > > > I had been discussing the same with GPFS support for past few months, and > > it seems to be that most of the time is being spent at cxiUXfer. They could > > not understand on why it is taking spending so much of time in cxiUXfer. I > > was seeing the same from perf top, and pagefaults. > > > > Below is snippet from what the support had said : > > > > ???????????????????????????? > > > > I searched all of the gpfsRead from trace and sort them by spending-time. > > Except 2 reads which need fetch data from nsd server, the slowest read is > > in the thread 72170. It took 112470.362 us. > > > > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum:?? 72165?????? 6.860911319 > > rdwr?????????????????? 141857.076 us + NSDIO > > > > trcrpt.2018-08-06_12.26.28.39794.lt15.trsum:?? 72170?????? 1.483947593 > > rdwr?????????????????? 112470.362 us + cxiUXfer > > > > trcrpt.2018-08-06_12.27.39.55538.lt15.trsum:?? 72165?????? 6.949042593 > > rdwr??????????????????? 88126.278 us + NSDIO > > > > trcrpt.2018-08-06_12.27.03.47706.lt15.trsum:?? 72156?????? 2.919334474 > > rdwr??????????????????? 81057.657 us + cxiUXfer > > > > trcrpt.2018-08-06_12.23.30.72745.lt15.trsum:?? 72154?????? 1.167484466 > > rdwr??????????????????? 76033.488 us + cxiUXfer > > > > trcrpt.2018-08-06_12.24.06.7508.lt15.trsum:?? 72187?????? 0.685237501 > > rdwr??????????????????? 70772.326 us + cxiUXfer > > > > trcrpt.2018-08-06_12.25.17.23989.lt15.trsum:?? 72193?????? 4.757996530 > > rdwr??????????????????? 70447.838 us + cxiUXfer > > > > > > I check each of the slow IO as above, and find they all spend much time in > > the function cxiUXfer. This function is used to copy data from kernel > > buffer to user buffer. I am not sure why it took so much time. This should > > be related to the pagefaults and pgfree you observed. Below is the trace > > data for thread 72170. > > > > > >??????????????????? 1.371477231? 72170 TRACE_VNODE: gpfs_f_rdwr enter: fP > > 0xFFFF882541649400 f_flags 0x8000 flags 0x8001 op 0 iovec > > 0xFFFF881F2AFB3E70 count 1 offset 0x168F30D dentry 0xFFFF887C0CC298C0 > > private 0xFFFF883F607175C0 iP 0xFFFF8823AA3CBFC0 name '410513.svs' > > > >?????????????? .... > > > >??????????????????? 1.371483547? 72170 TRACE_KSVFS: cachedReadFast exit: > > uio_resid 16777216 code 1 err 11 > > > >?????????????? .... > > > >??????????????????? 1.371498780? 72170 TRACE_KSVFS: kSFSReadFast: oiP > > 0xFFFFC90060B46740 offset 0x168F30D dataBufP FFFFC9003645A5A8 nDesc 64 buf > > 200043C0000 valid words 64 dirty words 0 blkOff 0 > > > >??????????????????? 1.371499035? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate begin ul 0xFFFFC900333F1A40 holdCount 0 > > ioType 0x2 inProg 0x15 > > > >??????????????????? 1.371500157? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate ul 0xFFFFC900333F1A40 holdCount 0 ioType 0x2 > > inProg 0x16 err 0 > > > >??????????????????? 1.371500606? 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 toIOBuf 0 offset 6877965 len > > 9899251 > > > >??????????????????? 1.371500793? 72170 TRACE_KSVFS: cxiUXfer: ndesc 0 skip > > dataAddrP 0x200043C0000 currOffset 0 currLen 262144 bufOffset 6877965 > > > >?????????????? .... > > > >??????????????????? 1.371505949? 72170 TRACE_KSVFS: cxiUXfer: ndesc 25 skip > > dataAddrP 0x2001AF80000 currOffset 6553600 currLen 262144 bufOffset 6877965 > > > >??????????????????? 1.371506236? 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > > currOffset 6815744 tmpLen 262144 dataAddrP 0x2001AFCF30D currLen 199923 > > pageOffset 781 pageLen 3315 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.373649823? 72170 TRACE_KSVFS: cxiUXfer: nDesc 27 > > currOffset 7077888 tmpLen 262144 dataAddrP 0x20027400000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.375158799? 72170 TRACE_KSVFS: cxiUXfer: nDesc 28 > > currOffset 7340032 tmpLen 262144 dataAddrP 0x20027440000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.376661566? 72170 TRACE_KSVFS: cxiUXfer: nDesc 29 > > currOffset 7602176 tmpLen 262144 dataAddrP 0x2002C180000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.377892653? 72170 TRACE_KSVFS: cxiUXfer: nDesc 30 > > currOffset 7864320 tmpLen 262144 dataAddrP 0x2002C1C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >?????????????? .... > > > >??????????????????? 1.471389843? 72170 TRACE_KSVFS: cxiUXfer: nDesc 62 > > currOffset 16252928 tmpLen 262144 dataAddrP 0x2001D2C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.471845629? 72170 TRACE_KSVFS: cxiUXfer: nDesc 63 > > currOffset 16515072 tmpLen 262144 dataAddrP 0x2003EC80000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.472417149? 72170 TRACE_KSVFS: cxiDetachIOBuffer: > > dataPtr 0x200043C0000 plP 0xFFFF887F7B90D600 > > > >??????????????????? 1.472417775? 72170 TRACE_LOCK: unlock_vfs: type Data, > > key 0000000000000004:000000001B1F24BF:0000000000000001 lock_mode have ro > > token xw lock_state old [ ro:27 ] new [ ro:26 ] holdCount now 27 > > > >??????????????????? 1.472418427? 72170 TRACE_LOCK: hash tab lookup vfs: > > found cP 0xFFFFC9005FC0CDE0 holdCount now 14 > > > >??????????????????? 1.472418592? 72170 TRACE_LOCK: lock_vfs: type Data key > > 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode want ro status > > valid token xw/xw lock_state [ ro:12 ] flags 0x0 holdCount 14 > > > >??????????????????? 1.472419842? 72170 TRACE_KSVFS: kSFSReadFast: oiP > > 0xFFFFC90060B46740 offset 0x2000000 dataBufP FFFFC9003643C908 nDesc 64 buf > > 38033480000 valid words 64 dirty words 0 blkOff 0 > > > >??????????????????? 1.472420029? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate begin ul 0xFFFFC9005FC0CF98 holdCount 0 > > ioType 0x2 inProg 0xC > > > >??????????????????? 1.472420187? 72170 TRACE_LOG: > > UpdateLogger::beginDataUpdate ul 0xFFFFC9005FC0CF98 holdCount 0 ioType 0x2 > > inProg 0xD err 0 > > > >??????????????????? 1.472420652? 72170 TRACE_KSVFS: cxiUXfer: nDesc 64 1st > > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 toIOBuf 0 offset 0 len 6877965 > > > >??????????????????? 1.472420936? 72170 TRACE_KSVFS: cxiUXfer: nDesc 0 > > currOffset 0 tmpLen 262144 dataAddrP 0x38033480000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.472824790? 72170 TRACE_KSVFS: cxiUXfer: nDesc 1 > > currOffset 262144 tmpLen 262144 dataAddrP 0x380334C0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.473243905? 72170 TRACE_KSVFS: cxiUXfer: nDesc 2 > > currOffset 524288 tmpLen 262144 dataAddrP 0x38024280000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >?????????????? .... > > > >??????????????????? 1.482949347? 72170 TRACE_KSVFS: cxiUXfer: nDesc 24 > > currOffset 6291456 tmpLen 262144 dataAddrP 0x38025E80000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483354265? 72170 TRACE_KSVFS: cxiUXfer: nDesc 25 > > currOffset 6553600 tmpLen 262144 dataAddrP 0x38025EC0000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483766631? 72170 TRACE_KSVFS: cxiUXfer: nDesc 26 > > currOffset 6815744 tmpLen 262144 dataAddrP 0x38003B00000 currLen 262144 > > pageOffset 0 pageLen 4096 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483943894? 72170 TRACE_KSVFS: cxiDetachIOBuffer: > > dataPtr 0x38033480000 plP 0xFFFF887F7B934320 > > > >??????????????????? 1.483944339? 72170 TRACE_LOCK: unlock_vfs: type Data, > > key 0000000000000004:000000001B1F24BF:0000000000000002 lock_mode have ro > > token xw lock_state old [ ro:14 ] new [ ro:13 ] holdCount now 14 > > > >??????????????????? 1.483944683? 72170 TRACE_BRL: brUnlockM: ofP > > 0xFFFFC90069346B68 inode 455025855 snap 0 handle 0xFFFFC9003637D020 range > > 0x168F30D-0x268F30C mode ro > > > >??????????????????? 1.483944985? 72170 TRACE_KSVFS: kSFSReadFast exit: > > uio_resid 0 err 0 > > > >??????????????????? 1.483945264? 72170 TRACE_LOCK: unlock_vfs_m: type > > Inode, key 305F105B9701E60A:000000001B1F24BF:0000000000000000 lock_mode > > have ro status valid token rs lock_state old [ ro:25 ] new [ ro:24 ] > > > >??????????????????? 1.483945423? 72170 TRACE_LOCK: unlock_vfs_m: cP > > 0xFFFFC90069346B68 holdCount 25 > > > >??????????????????? 1.483945624? 72170 TRACE_VNODE: gpfsRead exit: fast err > > 0 > > > >??????????????????? 1.483946831? 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > > 0xFFFFC90035E52F78 NotQuiesced vfsOp 2 > > > >??????????????????? 1.483946975? 72170 TRACE_KSVFS: ReleSG: sli 38 sgP > > 0xFFFFC90035E52F78 vfsOp 2 users 1-1 > > > >??????????????????? 1.483947116? 72170 TRACE_KSVFS: ReleaseDaemonSegAndSG: > > sli 38 count 2 needCleanup 0 > > > >??????????????????? 1.483947593? 72170 TRACE_VNODE: gpfs_f_rdwr exit: fP > > 0xFFFF882541649400 total_len 16777216 uio_resid 0 offset 0x268F30D rc 0 > > > > > > ??????????????????????????????????????????? > > > > > > > > Regards, > > Lohit > > > > On Sep 19, 2018, 3:11 PM -0400, Sven Oehme , wrote: > > > > the document primarily explains all performance specific knobs. general > > advice would be to longer set anything beside workerthreads, pagepool and > > filecache on 5.X systems as most other settings are no longer relevant > > (thats a client side statement) . thats is true until you hit strange > > workloads , which is why all the knobs are still there :-) > > > > sven > > > > > > On Wed, Sep 19, 2018 at 11:17 AM wrote: > > > >> Thanks Sven. > >> I will disable it completely and see how it behaves. > >> > >> Is this the presentation? > >> > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__files.gpfsug.org_presentations_2014_UG10-5FGPFS-5FPerformance-5FSession-5Fv10.pdf&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=-Y1ncRDmgOCfWDQaXj-OcbiM5gcs0LcllAShfNapHtI&e= > >> > >> I guess i read it, but it did not strike me at this situation. I will try > >> to read it again and see if i could make use of it. > >> > >> Regards, > >> Lohit > >> > >> On Sep 19, 2018, 2:12 PM -0400, Sven Oehme , wrote: > >> > >> seem like you never read my performance presentation from a few years ago > >> ;-) > >> > >> you can control this on a per node basis , either for all i/o : > >> > >>??? prefetchAggressiveness = X > >> > >> or individual for reads or writes : > >> > >>??? prefetchAggressivenessRead = X > >>??? prefetchAggressivenessWrite = X > >> > >> for a start i would turn it off completely via : > >> > >> mmchconfig prefetchAggressiveness=0 -I -N nodename > >> > >> that will turn it off only for that node and only until you restart the > >> node. > >> then see what happens > >> > >> sven > >> > >> > >> On Wed, Sep 19, 2018 at 11:07 AM wrote: > >> > >>> Thank you Sven. > >>> > >>> I mostly think it could be 1. or some other issue. > >>> I don?t think it could be 2. , because i can replicate this issue no > >>> matter what is the size of the dataset. It happens for few files that could > >>> easily fit in the page pool too. > >>> > >>> I do see a lot more page faults for 16M compared to 1M, so it could be > >>> related to many threads trying to compete for the same buffer space. > >>> > >>> I will try to take the trace with trace=io option and see if can find > >>> something. > >>> > >>> How do i turn of prefetching? Can i turn it off for a single > >>> node/client? > >>> > >>> Regards, > >>> Lohit > >>> > >>> On Sep 18, 2018, 5:23 PM -0400, Sven Oehme , wrote: > >>> > >>> Hi, > >>> > >>> taking a trace would tell for sure, but i suspect what you might be > >>> hitting one or even multiple issues which have similar negative performance > >>> impacts but different root causes. > >>> > >>> 1. this could be serialization around buffer locks. as larger your > >>> blocksize gets as larger is the amount of data one of this pagepool buffers > >>> will maintain, if there is a lot of concurrency on smaller amount of data > >>> more threads potentially compete for the same buffer lock to copy stuff in > >>> and out of a particular buffer, hence things go slower compared to the same > >>> amount of data spread across more buffers, each of smaller size. > >>> > >>> 2. your data set is small'ish, lets say a couple of time bigger than the > >>> pagepool and you random access it with multiple threads. what will happen > >>> is that because it doesn't fit into the cache it will be read from the > >>> backend. if multiple threads hit the same 16 mb block at once with multiple > >>> 4k random reads, it will read the whole 16mb block because it thinks it > >>> will benefit from it later on out of cache, but because it fully random the > >>> same happens with the next block and the next and so on and before you get > >>> back to this block it was pushed out of the cache because of lack of enough > >>> pagepool. > >>> > >>> i could think of multiple other scenarios , which is why its so hard to > >>> accurately benchmark an application because you will design a benchmark to > >>> test an application, but it actually almost always behaves different then > >>> you think it does :-) > >>> > >>> so best is to run the real application and see under which configuration > >>> it works best. > >>> > >>> you could also take a trace with trace=io and then look at > >>> > >>> TRACE_VNOP: READ: > >>> TRACE_VNOP: WRITE: > >>> > >>> and compare them to > >>> > >>> TRACE_IO: QIO: read > >>> TRACE_IO: QIO: write > >>> > >>> and see if the numbers summed up for both are somewhat equal. if > >>> TRACE_VNOP is significant smaller than TRACE_IO you most likely do more i/o > >>> than you should and turning prefetching off might actually make things > >>> faster . > >>> > >>> keep in mind i am no longer working for IBM so all i say might be > >>> obsolete by now, i no longer have access to the one and only truth aka the > >>> source code ... but if i am wrong i am sure somebody will point this out > >>> soon ;-) > >>> > >>> sven > >>> > >>> > >>> > >>> > >>> On Tue, Sep 18, 2018 at 10:31 AM wrote: > >>> > >>>> Hello All, > >>>> > >>>> This is a continuation to the previous discussion that i had with Sven. > >>>> However against what i had mentioned previously - i realize that this > >>>> is ?not? related to mmap, and i see it when doing random freads. > >>>> > >>>> I see that block-size of the filesystem matters when reading from Page > >>>> pool. > >>>> I see a major difference in performance when compared 1M to 16M, when > >>>> doing lot of random small freads with all of the data in pagepool. > >>>> > >>>> Performance for 1M is a magnitude ?more? than the performance that i > >>>> see for 16M. > >>>> > >>>> The GPFS that we have currently is : > >>>> Version : 5.0.1-0.5 > >>>> Filesystem version: 19.01 (5.0.1.0) > >>>> Block-size : 16M > >>>> > >>>> I had made the filesystem block-size to be 16M, thinking that i would > >>>> get the most performance for both random/sequential reads from 16M than the > >>>> smaller block-sizes. > >>>> With GPFS 5.0, i made use the 1024 sub-blocks instead of 32 and thus > >>>> not loose lot of storage space even with 16M. > >>>> I had run few benchmarks and i did see that 16M was performing better > >>>> ?when hitting storage/disks? with respect to bandwidth for > >>>> random/sequential on small/large reads. > >>>> > >>>> However, with this particular workload - where it freads a chunk of > >>>> data randomly from hundreds of files -> I see that the number of > >>>> page-faults increase with block-size and actually reduce the performance. > >>>> 1M performs a lot better than 16M, and may be i will get better > >>>> performance with less than 1M. > >>>> It gives the best performance when reading from local disk, with 4K > >>>> block size filesystem. > >>>> > >>>> What i mean by performance when it comes to this workload - is not the > >>>> bandwidth but the amount of time that it takes to do each iteration/read > >>>> batch of data. > >>>> > >>>> I figure what is happening is: > >>>> fread is trying to read a full block size of 16M - which is good in a > >>>> way, when it hits the hard disk. > >>>> But the application could be using just a small part of that 16M. Thus > >>>> when randomly reading(freads) lot of data of 16M chunk size - it is page > >>>> faulting a lot more and causing the performance to drop . > >>>> I could try to make the application do read instead of freads, but i > >>>> fear that could be bad too since it might be hitting the disk with a very > >>>> small block size and that is not good. > >>>> > >>>> With the way i see things now - > >>>> I believe it could be best if the application does random reads of > >>>> 4k/1M from pagepool but some how does 16M from rotating disks. > >>>> > >>>> I don?t see any way of doing the above other than following a different > >>>> approach where i create a filesystem with a smaller block size ( 1M or less > >>>> than 1M ), on SSDs as a tier. > >>>> > >>>> May i please ask for advise, if what i am understanding/seeing is right > >>>> and the best solution possible for the above scenario. > >>>> > >>>> Regards, > >>>> Lohit > >>>> > >>>> On Apr 11, 2018, 10:36 AM -0400, Lohit Valleru , > >>>> wrote: > >>>> > >>>> Hey Sven, > >>>> > >>>> This is regarding mmap issues and GPFS. > >>>> We had discussed previously of experimenting with GPFS 5. > >>>> > >>>> I now have upgraded all of compute nodes and NSD nodes to GPFS 5.0.0.2 > >>>> > >>>> I am yet to experiment with mmap performance, but before that - I am > >>>> seeing weird hangs with GPFS 5 and I think it could be related to mmap. > >>>> > >>>> Have you seen GPFS ever hang on this syscall? > >>>> [Tue Apr 10 04:20:13 2018] [] > >>>> _ZN10gpfsNode_t8mmapLockEiiPKj+0xb5/0x140 [mmfs26] > >>>> > >>>> I see the above ,when kernel hangs and throws out a series of trace > >>>> calls. > >>>> > >>>> I somehow think the above trace is related to processes hanging on GPFS > >>>> forever. There are no errors in GPFS however. > >>>> > >>>> Also, I think the above happens only when the mmap threads go above a > >>>> particular number. > >>>> > >>>> We had faced a similar issue in 4.2.3 and it was resolved in a patch to > >>>> 4.2.3.2 . At that time , the issue happened when mmap threads go more than > >>>> worker1threads. According to the ticket - it was a mmap race condition that > >>>> GPFS was not handling well. > >>>> > >>>> I am not sure if this issue is a repeat and I am yet to isolate the > >>>> incident and test with increasing number of mmap threads. > >>>> > >>>> I am not 100 percent sure if this is related to mmap yet but just > >>>> wanted to ask you if you have seen anything like above. > >>>> > >>>> Thanks, > >>>> > >>>> Lohit > >>>> > >>>> On Feb 22, 2018, 3:59 PM -0500, Sven Oehme , wrote: > >>>> > >>>> Hi Lohit, > >>>> > >>>> i am working with ray on a mmap performance improvement right now, > >>>> which most likely has the same root cause as yours , see --> > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_2018-2DJanuary_004411.html&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=AUEL847F_a-j6Y7t1fDMj4j33vLqvI6XrrNCVS5pUyA&e= > >>>> the thread above is silent after a couple of back and rorth, but ray > >>>> and i have active communication in the background and will repost as soon > >>>> as there is something new to share. > >>>> i am happy to look at this issue after we finish with ray's workload if > >>>> there is something missing, but first let's finish his, get you try the > >>>> same fix and see if there is something missing. > >>>> > >>>> btw. if people would share their use of MMAP , what applications they > >>>> use (home grown, just use lmdb which uses mmap under the cover, etc) please > >>>> let me know so i get a better picture on how wide the usage is with GPFS. i > >>>> know a lot of the ML/DL workloads are using it, but i would like to know > >>>> what else is out there i might not think about. feel free to drop me a > >>>> personal note, i might not reply to it right away, but eventually. > >>>> > >>>> thx. sven > >>>> > >>>> > >>>> On Thu, Feb 22, 2018 at 12:33 PM wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> I wanted to know, how does mmap interact with GPFS pagepool with > >>>>> respect to filesystem block-size? > >>>>> Does the efficiency depend on the mmap read size and the block-size of > >>>>> the filesystem even if all the data is cached in pagepool? > >>>>> > >>>>> GPFS 4.2.3.2 and CentOS7. > >>>>> > >>>>> Here is what i observed: > >>>>> > >>>>> I was testing a user script that uses mmap to read from 100M to 500MB > >>>>> files. > >>>>> > >>>>> The above files are stored on 3 different filesystems. > >>>>> > >>>>> Compute nodes - 10G pagepool and 5G seqdiscardthreshold. > >>>>> > >>>>> 1. 4M block size GPFS filesystem, with separate metadata and data. > >>>>> Data on Near line and metadata on SSDs > >>>>> 2. 1M block size GPFS filesystem as a AFM cache cluster, "with all the > >>>>> required files fully cached" from the above GPFS cluster as home. Data and > >>>>> Metadata together on SSDs > >>>>> 3. 16M block size GPFS filesystem, with separate metadata and data. > >>>>> Data on Near line and metadata on SSDs > >>>>> > >>>>> When i run the script first time for ?each" filesystem: > >>>>> I see that GPFS reads from the files, and caches into the pagepool as > >>>>> it reads, from mmdiag -- iohist > >>>>> > >>>>> When i run the second time, i see that there are no IO requests from > >>>>> the compute node to GPFS NSD servers, which is expected since all the data > >>>>> from the 3 filesystems is cached. > >>>>> > >>>>> However - the time taken for the script to run for the files in the 3 > >>>>> different filesystems is different - although i know that they are just > >>>>> "mmapping"/reading from pagepool/cache and not from disk. > >>>>> > >>>>> Here is the difference in time, for IO just from pagepool: > >>>>> > >>>>> 20s 4M block size > >>>>> 15s 1M block size > >>>>> 40S 16M block size. > >>>>> > >>>>> Why do i see a difference when trying to mmap reads from different > >>>>> block-size filesystems, although i see that the IO requests are not hitting > >>>>> disks and just the pagepool? > >>>>> > >>>>> I am willing to share the strace output and mmdiag outputs if needed. > >>>>> > >>>>> Thanks, > >>>>> Lohit > >>>>> > >>>>> _______________________________________________ > >>>>> gpfsug-discuss mailing list > >>>>> gpfsug-discuss at spectrumscale.org > >>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>>> _______________________________________________ > >>>> gpfsug-discuss mailing list > >>>> gpfsug-discuss at spectrumscale.org > >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>> > >>> _______________________________________________ > >>> gpfsug-discuss mailing list > >>> gpfsug-discuss at spectrumscale.org > >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >>> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >> > >> _______________________________________________ > >> gpfsug-discuss mailing list > >> gpfsug-discuss at spectrumscale.org > >> https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > >> > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > > _______________________________________________ > > gpfsug-discuss mailing list > > gpfsug-discuss at spectrumscale.org > > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=C9X8xNkG_lwP_-eFHTGejw&r=dOGWbX02X0lU3_xWy7sXmdsAk4pvqANuwZ0j-sV6OEo&m=ZkmRYnq8ro8C7ccXHnRvIVN4PzFuuQz-VKI8RiuM0ow&s=kDfJQ7W_JgLnvD_F6kwpIEFg_9j-Ain9f1uvMKFJD6s&e= > > > End of gpfsug-discuss Digest, Vol 81, Issue 43 > ********************************************** > This message is for the recipient?s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From WPeters at ATPCO.NET Wed Nov 20 18:16:23 2019 From: WPeters at ATPCO.NET (Peters, Bill) Date: Wed, 20 Nov 2019 18:16:23 +0000 Subject: [gpfsug-discuss] introduction Message-ID: Hello, The welcome email said I should introduce myself to the group. I'm Bill Peters, a Linux engineer at ATPCO (Airline Tariff Publishing Company) located in Dulles, Virginia, USA. We process airline fare data. I've was recently introduced to Spectrum Scale because we are doing a proof of concept to see if we can move some of our workload onto virtual machines running in IBM's zVM. Our x86 systems use Veritas for the network filesystem and since Veritas doesn't support the s390 arcitecture, we are using Spectrum Scale. So far it's been great, much easier to understand than Veritas. We're not doing anything too complicated. The disks are DASD on SSD. We have 3 clusters with sharing between them. At this point I expect the POC to be successful so I will probably be working with Spectrum Scale into the future. The only question I have so far is read speeds seem to be slower than Veritas. It's not slow enough to be a real problem, but if anyone has a suggestion for speeding up reads, I'd love to hear it. Other than that, I'll probably just be lurking on the email list. Thanks, -Bill -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Nov 20 19:56:50 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 20 Nov 2019 20:56:50 +0100 Subject: [gpfsug-discuss] introduction In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Wed Nov 20 20:06:53 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Wed, 20 Nov 2019 21:06:53 +0100 Subject: [gpfsug-discuss] introduction In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From mweil at wustl.edu Wed Nov 20 20:44:05 2019 From: mweil at wustl.edu (Weil, Matthew) Date: Wed, 20 Nov 2019 20:44:05 +0000 Subject: [gpfsug-discuss] fsstruct and FSCK IBM case TS002836092 Message-ID: <5889a4ef-0c62-8ee8-7347-a7bb6b3acb74@wustl.edu> Ulf Troppens, Would it be possible for you to engage and assist with this case? Thanks Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: From gterryc at vmsupport.com Fri Nov 22 00:24:02 2019 From: gterryc at vmsupport.com (George Terry) Date: Thu, 21 Nov 2019 18:24:02 -0600 Subject: [gpfsug-discuss] Compatibility question Message-ID: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George -------------- next part -------------- An HTML attachment was scrubbed... URL: From luke.raimbach at googlemail.com Fri Nov 22 01:12:27 2019 From: luke.raimbach at googlemail.com (Luke Raimbach) Date: Thu, 21 Nov 2019 19:12:27 -0600 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: Message-ID: That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, wrote: > Hello, > > I have a question about compatibility between two different versions. I > have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are > clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to > application's issues I cannot update the client nodes. So, my question is, > can i have a cluster with client nodes with 3.5.0.7 version and the NSD > nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? > > Thanks for the support. > > George > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > -------------- next part -------------- An HTML attachment was scrubbed... URL: From knop at us.ibm.com Fri Nov 22 04:49:12 2019 From: knop at us.ibm.com (Felipe Knop) Date: Fri, 22 Nov 2019 04:49:12 +0000 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: From S.J.Thompson at bham.ac.uk Fri Nov 22 14:10:01 2019 From: S.J.Thompson at bham.ac.uk (Simon Thompson) Date: Fri, 22 Nov 2019 14:10:01 +0000 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: Message-ID: My guess would be rhel6 clients. Though rhel6 should be able to run a 4.x release and then you are N-1. Our one and only remaining centos6 box, well we unceremoniously kicked that out of the cluster and made it use NFS. Simon From: on behalf of "luke.raimbach at googlemail.com" Reply to: "gpfsug-discuss at spectrumscale.org" Date: Friday, 22 November 2019 at 01:12 To: "gpfsug-discuss at spectrumscale.org" Subject: Re: [gpfsug-discuss] Compatibility question That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, > wrote: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: From sadaniel at us.ibm.com Fri Nov 22 17:01:09 2019 From: sadaniel at us.ibm.com (Steven Daniels) Date: Fri, 22 Nov 2019 10:01:09 -0700 Subject: [gpfsug-discuss] Compatibility question In-Reply-To: References: , Message-ID: Felipe, You are correct. The code is restricted to 4.2.2 or higher. If there is an ESS storage cluster in the environment then the recommendation I was told is 4.2.3.7 or higher. Luke, If your clients are in the same cluster then you need to be careful with the setting to minReleaseLevel, Exactly as it sounds, it will restrict membership to this level or higher. I have a client running with filesystem version 3.5.0.7 in several clusters. We just recently upgraded their ESS systems to the current release so I can verify that the file system format will carry fine. If you have ESS at 5.3.3 or higher, then the GNR recommendation is 4.2.3.7 or higher for the clients (my client can't go to v5 due to the RHEL version requirement) and they must be in a remote cluster. In my case we upgraded an ESS storage cluster to 4.1.1 to 5.0.3.3 (ESS v5.3.4.1) and for GNR we had to update this cluster to LATEST for minReleaseLevel which is 5.0.3. The client cluster is running 4.2.3.18 which is as Felipe says is N-1 and minReleaseLevel is set to 4.2.3.7 This client cluster also mounts from another storage cluster which is constrained to run at 4.1.1-8. This makes the client cluster at N+1 from its relationship with this cluster. All three clusters have minReleaseLevel set to LATEST (4.1.0, 4.2.3.7,5.0.3 , respectively). These changes would not work in a single cluster as all nodes would be restricted to the 4.1.0 or higher and the ESS would not be able to co-exist since it requires the 5.0 level. We have updated the file systems in the 4.1.1-8 cluster from 3.5.0.7 to 4.1.1 and we updated the ESS filesystems from 3.5.0.7 to 5.0.3 (compat mode) The changes all went well. Hopefully this helps you understand the nuance that is whether the clients and NSDs are in a single cluster or different clusters. The rules are slightly but importantly different. Also, when updating the filesystem versions make sure you use the "compat" flag if you have any remote clusters. If we would have run this with the "full" flag on the ESS storage cluster, it would have prevented the 4.2.3.18 client cluster from mounting its file systems. thanks, Steve Steven A. Daniels Cross-brand Client Architect Senior Certified IT Specialist National Programs Fax and Voice: 3038101229 sadaniel at us.ibm.com http://www.ibm.com From: "Felipe Knop" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 11/21/2019 09:49 PM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question Sent by: gpfsug-discuss-bounces at spectrumscale.org George, Coexistence either in the same cluster or across remote clusters is only supported with versions "N" and "N-1". 5.0.3 and 3.5 are "N" and "N-3", and this is not supported. If I recall, there is even logic in place that will prevent 5.x nodes from accepting connections from nodes running below 4.2 . Felipe ---- Felipe Knop knop at us.ibm.com GPFS Development and Security IBM Systems IBM Building 008 2455 South Rd, Poughkeepsie, NY 12601 (845) 433-9314 T/L 293-9314 ----- Original message ----- From: Luke Raimbach Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question Date: Thu, Nov 21, 2019 8:12 PM That would make my eyes water and maybe fall out. Yours might too of you try it. Be very careful with v3 upgrades, especially going up to v5 directly. You will certainly run into RPC incompatibility problems with the v3 and v5 nodes running concurrently. Staying within the supported envelope of "one major version" difference will ease the raising of support tickets with IBM if you run into problems. Why can't you upgrade the clients? On Thu, 21 Nov 2019, 18:24 George Terry, wrote: Hello, I have a question about compatibility between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3. Due to application's issues I cannot update the client nodes. So, my question is, can i have a cluster with client nodes with 3.5.0.7 version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23) version? Thanks for the support. George _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6Ym4Q&m=sHhNGNTp4CJIVbXT4t8ZvNT_mETKfXBn7XTvNDAYF7s&s=9ialbzpuZMc8DZzXtU_nqjgQ9BPsHP3DQdpHa0nKQuc&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/jpeg Size: 4919 bytes Desc: not available URL: From werner at usit.uio.no Tue Nov 26 06:25:08 2019 From: werner at usit.uio.no (Morten Werner Forsbring) Date: Tue, 26 Nov 2019 07:25:08 +0100 Subject: [gpfsug-discuss] New subscriber Message-ID: <87blsz6tob.fsf@sue.uio.no> Hi. My name is Morten Werner Forsbring and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with different services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Best regards! - Werner From helge.hauglin at usit.uio.no Tue Nov 26 10:25:21 2019 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 26 Nov 2019 11:25:21 +0100 Subject: [gpfsug-discuss] New member: Helge Hauglin Message-ID: Hi, My name is Helge Hauglin, and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Previous storage experience: NetApp and Hitachi Vantara (Hitachi NAS and block storage). Best regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From christophe.darras at atempo.com Tue Nov 26 11:21:29 2019 From: christophe.darras at atempo.com (Christophe Darras) Date: Tue, 26 Nov 2019 11:21:29 +0000 Subject: [gpfsug-discuss] New member: Helge Hauglin In-Reply-To: References: Message-ID: Hello Helge, Thanks for the introduction. Might you need some help to migrate your unstructured data from HDS and NetApp to GPFS, pls let us know, Kindest Regards, Christophe DARRAS ? Head of North Europe, Mobile : +44 7?555 99 3529 christophe.darras at atempo.com Atempo ltd N?1 EUROPEAN SOFTWARE VENDOR FOR DATA PROTECTION? ? ATEMPO.COM ? ? ? ? -----Original Message----- From: gpfsug-discuss-bounces at spectrumscale.org On Behalf Of Helge Hauglin Sent: mardi 26 novembre 2019 10:25 To: gpfsug-discuss at spectrumscale.org Subject: [gpfsug-discuss] New member: Helge Hauglin Hi, My name is Helge Hauglin, and I work for the University of Oslo, Norway. We are in the process of installing Spectrum Scale for use with services for our researchers (general file storage, visualization, HPC, ..). We will probably also implement Spectrum Scale as the next general file service for the university. Previous storage experience: NetApp and Hitachi Vantara (Hitachi NAS and block storage). Best regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss From helge.hauglin at usit.uio.no Tue Nov 26 13:21:10 2019 From: helge.hauglin at usit.uio.no (Helge Hauglin) Date: Tue, 26 Nov 2019 14:21:10 +0100 Subject: [gpfsug-discuss] New member: Helge Hauglin In-Reply-To: (Christophe Darras's message of "Tue, 26 Nov 2019 11:21:29 +0000") References: Message-ID: Hi Christophe, Thanks for the information, but we'll use our in-house expertise to move data to the ESS. -- Regards, Helge Hauglin ---------------------------------------------------------------- Mr. Helge Hauglin, Senior Engineer System administrator Center for Information Technology, University of Oslo, Norway From mark.bergman at uphs.upenn.edu Tue Nov 26 23:58:02 2019 From: mark.bergman at uphs.upenn.edu (mark.bergman at uphs.upenn.edu) Date: Tue, 26 Nov 2019 18:58:02 -0500 Subject: [gpfsug-discuss] python package installs fail due to attribute copy error (GPFS 5.0.2) Message-ID: <32070-1574812682.257155@ttRP.UhP3.tHtF> We're running GPFS 5.0.2 (DSS-G 2.2a, in the process of upgrading to 2.4b/GPFS 5.0.3.2) with NFSv4 ACLs and no explicit ACLs on the root or dependent filesets. While installing python packages within a python virtual env stored under GPFS, we get failures with "permission denied" on the destination while trying to copy attributes. The problem happens to root & end-users, and is not related to the directory permissions, ownership, or group, and is consistent and repeatable. For example, trying to use 'conda' to create a virtual environment results in the following message (severely trimmed): # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): in clone_env shutil.copystat(src, dst) in copystat _copyxattr(src, dst, follow_symlinks=follow) in _copyxattr os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) PermissionError: [Errno 13] Permission denied: '/gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6/lib/python3.5/site-packages/dm/neuralnet/__pycache__/__init__.cpython-35.pyc' `$ python/anaconda/3/bin/conda create --prefix /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6 --clone /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6` Note that the destination directory path is created, and the destination file (__init__.cython-35.pyc) is created...the exception is thrown when python tries to apply the ACLs from the source to the destination. The problem seems to be simiar to this: https://github.com/conda/conda/issues/5251 but our version of create.py is newer and already has the suggested fix. or similar to this: https://bugs.python.org/issue24538 that's affecting panfs. We've got a work-around using symlinks, but it's not something for average users. Has anyone seen this issue? Thanks, Mark "about to post to various python fora" Bergman From b.cregan at imperial.ac.uk Thu Nov 28 09:43:55 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 09:43:55 +0000 Subject: [gpfsug-discuss] Compression question Message-ID: Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 09:49:32 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 09:49:32 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Thu Nov 28 10:05:23 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 10:05:23 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 10:13:35 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 10:13:35 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 12:21:00 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 12:21:00 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Thu Nov 28 12:30:25 2019 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 28 Nov 2019 12:30:25 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 12:49:01 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 12:49:01 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191283.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191284.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191285.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Thu Nov 28 12:57:37 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 12:57:37 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , , Message-ID: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com [https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png] [https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png] [https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png] ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards [cid:15749424191280] [IBM Spectrum Scale] * * * Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com [cid:15749424191282] IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191280.png Type: image/png Size: 1134 bytes Desc: Image.15749424191280.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191281.png Type: image/png Size: 6645 bytes Desc: Image.15749424191281.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191282.png Type: image/png Size: 1134 bytes Desc: Image.15749424191282.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png Type: image/png Size: 1641 bytes Desc: Image.f7e9a6a7-c4ab-4a2c-a979-4513774aa054.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png Type: image/png Size: 17519 bytes Desc: Image.dbd2fd87-b9f2-4b15-903f-c74c7b7298d3.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png Type: image/png Size: 1641 bytes Desc: Image.ba38b5d6-12d6-476a-bcd8-7d647bfab5b3.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png Type: image/png Size: 17519 bytes Desc: Image.e34493c2-d35b-4fff-a6df-ce87e2ae5ed4.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From luis.bolinches at fi.ibm.com Thu Nov 28 13:00:22 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 13:00:22 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: Message-ID: Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers > On 28. Nov 2019, at 14.57, Cregan, Bob wrote: > > ? > Hi > Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? > > After compression of an existing file we get in the snap > > -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf > file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf > metadata replication: 2 max 2 > data replication: 1 max 3 > immutable: no > appendOnly: no > flags: > storage pool name: sata1 > fileset name: userdirs > snapshot name: @GMT-2019.11.27-19.30.14 > creation time: Tue Mar 5 16:16:40 2019 > Misc attributes: ARCHIVE > Encrypted: no > > and the original file is definitely compressed. > > -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf > file name: UserGuide_13.06.pdf > metadata replication: 2 max 2 > data replication: 1 max 3 > immutable: no > appendOnly: no > flags: > storage pool name: sata1 > fileset name: userdirs > snapshot name: > creation time: Tue Mar 5 16:16:40 2019 > Misc attributes: ARCHIVE COMPRESSION (library z) > Encrypted: no > > Bob > > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger > Sent: 28 November 2019 12:30 > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Compression question > > Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial > > > Alexander, > > Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? > Daniel > > _________________________________________________________ > Daniel Kidger > IBM Technical Sales Specialist > Spectrum Scale, Spectrum NAS and IBM Cloud Object Store > > +44-(0)7818 522 266 > daniel.kidger at uk.ibm.com > > > > > ----- Original message ----- > From: "Alexander Wolf" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 12:21 > > I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). > > Mit freundlichen Gr??en / Kind regards > > > > > > Dr. Alexander Wolf-Reber > Spectrum Scale Release Lead Architect > Department M069 / Spectrum Scale Software Development > > +49-160-90540880 > a.wolf-reber at de.ibm.com > > IBM Data Privacy Statement > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > ----- Original message ----- > From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 11:47 > > Hi > > Same principle COW. The data blocks do not get modified. > -- > Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions > Luis Bolinches > Consultant IT Specialist > Mobile Phone: +358503112585 > https://www.youracclaim.com/user/luis-bolinches > > "If you always give you will always have" -- Anonymous > > > > ----- Original message ----- > From: "Cregan, Bob" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: gpfsug main discussion list > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 12:06 PM > > Just to clarify this is SS compression, so > > mmchattr --compression yes > > or an ILM equivalent > > So not a regular modification. > > Bob > > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > > > > From: Cregan, Bob > Sent: 28 November 2019 09:43 > To: gpfsug main discussion list > Subject: Compression question > > Hi All, > Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. > > What happens to the snapshot when a file is compressed in SS? The logic as I see it is > > ####### In a non compressed situation ############### > > 1) create a file, > 2) create a snapshot. > 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. > > ######In a compressed situation ############ > > 1) create a file, > 2) create a snapshot. > 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. > > You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. > Thanks > > Bob > > > Bob Cregan > HPC Systems Analyst > Information & Communication Technologies > Imperial College London, > South Kensington Campus London, SW7 2AZ > T: 07712388129 > E: b.cregan at imperial.ac.uk > W: www.imperial.ac.uk/ict/rcs > > @imperialRCS @imperialRSE > > > > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > Unless stated otherwise above: > IBM United Kingdom Limited - Registered in England and Wales with number 741598. > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From b.cregan at imperial.ac.uk Thu Nov 28 13:05:43 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Thu, 28 Nov 2019 13:05:43 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: Hi 5.0.2 with a hotfix for a lz4 compression bug Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Luis Bolinches Sent: 28 November 2019 13:00 To: gpfsug main discussion list Subject: Re: [gpfsug-discuss] Compression question Caution - This email from luis.bolinches at fi.ibm.com originated outside Imperial Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob wrote: ? Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com [https://images.youracclaim.com/images/c49300ae-d13e-4071-90f5-15f59d199c9e/IBM%2BVolunteers%2BGold%2Bv6.png] [https://images.youracclaim.com/images/f2539224-f951-46b4-b376-b88f21c2be98/IBM-Selling-Certification---Level-1.png] [https://images.youracclaim.com/images/ea52b12f-97ac-4e72-8d24-b0ced8054e7d/Storage%2BTechnical%2BV1.png] ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards * * * Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From A.Wolf-Reber at de.ibm.com Thu Nov 28 14:02:46 2019 From: A.Wolf-Reber at de.ibm.com (Alexander Wolf) Date: Thu, 28 Nov 2019 14:02:46 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191286.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191287.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.15749424191288.png Type: image/png Size: 1134 bytes Desc: not available URL: From luis.bolinches at fi.ibm.com Thu Nov 28 14:07:27 2019 From: luis.bolinches at fi.ibm.com (Luis Bolinches) Date: Thu, 28 Nov 2019 14:07:27 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: Message-ID: That is the fix. Before it was an error. So blocks that are pointed from active get compressed. Ones that are no longer linked are not modified. -- Cheers > On 28. Nov 2019, at 16.03, Alexander Wolf wrote: > > ? > I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. > > Mit freundlichen Gr??en / Kind regards > > > > > > Dr. Alexander Wolf-Reber > Spectrum Scale Release Lead Architect > Department M069 / Spectrum Scale Software Development > > +49-160-90540880 > a.wolf-reber at de.ibm.com > > IBM Data Privacy Statement > IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp > Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 > > > > ----- Original message ----- > From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org > To: "gpfsug main discussion list" > Cc: > Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question > Date: Thu, Nov 28, 2019 14:00 > > Which version are you running? > > I was involved on a big for compressed file sets and snapshots that were related to what you see. > > -- > Cheers > >> >>> On 28. Nov 2019, at 14.57, Cregan, Bob wrote: >>> >> ? >> Hi >> Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? >> >> After compression of an existing file we get in the snap >> >> -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf >> file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf >> metadata replication: 2 max 2 >> data replication: 1 max 3 >> immutable: no >> appendOnly: no >> flags: >> storage pool name: sata1 >> fileset name: userdirs >> snapshot name: @GMT-2019.11.27-19.30.14 >> creation time: Tue Mar 5 16:16:40 2019 >> Misc attributes: ARCHIVE >> Encrypted: no >> >> and the original file is definitely compressed. >> >> -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf >> file name: UserGuide_13.06.pdf >> metadata replication: 2 max 2 >> data replication: 1 max 3 >> immutable: no >> appendOnly: no >> flags: >> storage pool name: sata1 >> fileset name: userdirs >> snapshot name: >> creation time: Tue Mar 5 16:16:40 2019 >> Misc attributes: ARCHIVE COMPRESSION (library z) >> Encrypted: no >> Bob >> >> >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> >> >> From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger >> Sent: 28 November 2019 12:30 >> To: gpfsug-discuss at spectrumscale.org >> Cc: gpfsug-discuss at spectrumscale.org >> Subject: Re: [gpfsug-discuss] Compression question >> >> Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial >> >> >> Alexander, >> >> Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? >> Daniel >> >> _________________________________________________________ >> Daniel Kidger >> IBM Technical Sales Specialist >> Spectrum Scale, Spectrum NAS and IBM Cloud Object Store >> >> +44-(0)7818 522 266 >> daniel.kidger at uk.ibm.com >> >> >> >> >> ----- Original message ----- >> From: "Alexander Wolf" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 12:21 >> >> I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). >> >> Mit freundlichen Gr??en / Kind regards >> >> >> >> >> >> >> >> >> Dr. Alexander Wolf-Reber >> Spectrum Scale Release Lead Architect >> Department M069 / Spectrum Scale Software Development >> >> +49-160-90540880 >> a.wolf-reber at de.ibm.com >> >> IBM Data Privacy Statement >> IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp >> Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 >> >> >> >> ----- Original message ----- >> From: "Luis Bolinches" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug-discuss at spectrumscale.org >> Cc: gpfsug-discuss at spectrumscale.org >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 11:47 >> >> Hi >> >> Same principle COW. The data blocks do not get modified. >> -- >> Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions >> Luis Bolinches >> Consultant IT Specialist >> Mobile Phone: +358503112585 >> https://www.youracclaim.com/user/luis-bolinches >> >> "If you always give you will always have" -- Anonymous >> >> >> >> ----- Original message ----- >> From: "Cregan, Bob" >> Sent by: gpfsug-discuss-bounces at spectrumscale.org >> To: gpfsug main discussion list >> Cc: >> Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question >> Date: Thu, Nov 28, 2019 12:06 PM >> >> Just to clarify this is SS compression, so >> >> mmchattr --compression yes >> >> or an ILM equivalent >> >> So not a regular modification. >> >> Bob >> >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> >> From: Cregan, Bob >> Sent: 28 November 2019 09:43 >> To: gpfsug main discussion list >> Subject: Compression question >> >> Hi All, >> Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. >> >> What happens to the snapshot when a file is compressed in SS? The logic as I see it is >> >> ####### In a non compressed situation ############### >> >> 1) create a file, >> 2) create a snapshot. >> 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. >> >> ######In a compressed situation ############ >> >> 1) create a file, >> 2) create a snapshot. >> 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. >> >> You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. >> Thanks >> >> Bob >> >> >> Bob Cregan >> HPC Systems Analyst >> Information & Communication Technologies >> Imperial College London, >> South Kensington Campus London, SW7 2AZ >> T: 07712388129 >> E: b.cregan at imperial.ac.uk >> W: www.imperial.ac.uk/ict/rcs >> >> >> @imperialRCS @imperialRSE >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> Ellei edell? ole toisin mainittu: / Unless stated otherwise above: >> Oy IBM Finland Ab >> PL 265, 00101 Helsinki, Finland >> Business ID, Y-tunnus: 0195876-3 >> Registered in Finland >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> Unless stated otherwise above: >> IBM United Kingdom Limited - Registered in England and Wales with number 741598. >> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU >> > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: > Oy IBM Finland Ab > PL 265, 00101 Helsinki, Finland > Business ID, Y-tunnus: 0195876-3 > Registered in Finland > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > > Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland -------------- next part -------------- An HTML attachment was scrubbed... URL: From olaf.weiser at de.ibm.com Thu Nov 28 15:00:56 2019 From: olaf.weiser at de.ibm.com (Olaf Weiser) Date: Thu, 28 Nov 2019 08:00:56 -0700 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 1134 bytes Desc: not available URL: From daniel.kidger at uk.ibm.com Thu Nov 28 15:28:28 2019 From: daniel.kidger at uk.ibm.com (Daniel Kidger) Date: Thu, 28 Nov 2019 15:28:28 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB17BD4DEB177B80052596E872584C0.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB18B14DEB187280052596E872584C0.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image._4_DEB19CD4DEB180B40052596E872584C0.png Type: image/png Size: 1134 bytes Desc: not available URL: From scale at us.ibm.com Thu Nov 28 17:09:37 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 28 Nov 2019 22:39:37 +0530 Subject: [gpfsug-discuss] =?utf-8?q?python_package_installs_fail_due_to_at?= =?utf-8?q?tribute_copy=09error_=28GPFS_5=2E0=2E2=29?= In-Reply-To: <32070-1574812682.257155@ttRP.UhP3.tHtF> References: <32070-1574812682.257155@ttRP.UhP3.tHtF> Message-ID: Hi Diane, Can you please help customer with the below issue. Or else can you point me to the right folks who can help here. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: mark.bergman at uphs.upenn.edu To: gpfsug main discussion list Date: 27-11-2019 05:47 Subject: [EXTERNAL] [gpfsug-discuss] python package installs fail due to attribute copy error (GPFS 5.0.2) Sent by: gpfsug-discuss-bounces at spectrumscale.org We're running GPFS 5.0.2 (DSS-G 2.2a, in the process of upgrading to 2.4b/GPFS 5.0.3.2) with NFSv4 ACLs and no explicit ACLs on the root or dependent filesets. While installing python packages within a python virtual env stored under GPFS, we get failures with "permission denied" on the destination while trying to copy attributes. The problem happens to root & end-users, and is not related to the directory permissions, ownership, or group, and is consistent and repeatable. For example, trying to use 'conda' to create a virtual environment results in the following message (severely trimmed): # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<< Traceback (most recent call last): in clone_env shutil.copystat(src, dst) in copystat _copyxattr(src, dst, follow_symlinks=follow) in _copyxattr os.setxattr(dst, name, value, follow_symlinks=follow_symlinks) PermissionError: [Errno 13] Permission denied: '/gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6/lib/python3.5/site-packages/dm/neuralnet/__pycache__/__init__.cpython-35.pyc' `$ python/anaconda/3/bin/conda create --prefix /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6 --clone /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6` Note that the destination directory path is created, and the destination file (__init__.cython-35.pyc) is created...the exception is thrown when python tries to apply the ACLs from the source to the destination. The problem seems to be simiar to this: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_conda_conda_issues_5251&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=8cqECGE7aQzVZwSOUDWjUbgMiyWLzpVqThnCrhMXwGc&e= but our version of create.py is newer and already has the suggested fix. or similar to this: https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue24538&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=Zyfy_KMaSuhXSmm5sUTBMiRqEiaIfD4BLSyRfpNjQkY&e= that's affecting panfs. We've got a work-around using symlinks, but it's not something for average users. Has anyone seen this issue? Thanks, Mark "about to post to various python fora" Bergman _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=kHkKLF_E6f-HbrmIODOa1m064mJD_5IUa4winUqTQOE&s=uVXOGz3UqVUU4d3Ov4e4-8wLRnxTsNtRPOv_iWcmOjM&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: From scale at us.ibm.com Thu Nov 28 17:22:43 2019 From: scale at us.ibm.com (IBM Spectrum Scale) Date: Thu, 28 Nov 2019 22:52:43 +0530 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , Message-ID: Hai Zhong, Can you please help the customer with their compression related query. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. From: "Daniel Kidger" To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 28-11-2019 20:58 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). And likewise for deeper (older) snapshots. sDaniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Olaf Weiser" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA (of course.. only as long nobody changed the file since the snapshot was taken) From: "Alexander Wolf" To: gpfsug-discuss at spectrumscale.org Date: 11/28/2019 07:03 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Gr??en / Kind regards IBM Spectrum Scale Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug main discussion list" Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 14:00 Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob wrote: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Daniel Kidger Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: Re: [gpfsug-discuss] Compression question |------------------------------------------------------------------------------| |Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial| | | |------------------------------------------------------------------------------| Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Alexander Wolf" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=IbxtjdkPAM2Sbon4Lbbi4w&m=6F5tNsgC3a8tnTmVCVxGFJgNmWLUYm_MUO-zYDgeQ5o&s=F1aJcNbQxqosWD5WimhGS19MpeszOyIp9CxuUib-c2Q&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: graycol.gif Type: image/gif Size: 105 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14065652.gif Type: image/gif Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 14488338.gif Type: image/gif Size: 6645 bytes Desc: not available URL: From zhouhz at cn.ibm.com Fri Nov 29 01:55:31 2019 From: zhouhz at cn.ibm.com (Hai Zhong HZ Zhou) Date: Fri, 29 Nov 2019 01:55:31 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361973.png Type: image/png Size: 1134 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361974.png Type: image/png Size: 6645 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361975.png Type: image/png Size: 1134 bytes Desc: not available URL: From b.cregan at imperial.ac.uk Fri Nov 29 07:22:56 2019 From: b.cregan at imperial.ac.uk (Cregan, Bob) Date: Fri, 29 Nov 2019 07:22:56 +0000 Subject: [gpfsug-discuss] Compression question In-Reply-To: References: , , , Message-ID: Hi Thanks very much everyone for this. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs [1505984389175_twitter.png] @imperialRCS @imperialRSE [1505983882959_Imperial-RCS.png] ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org on behalf of Hai Zhong HZ Zhou Sent: 29 November 2019 01:55 To: IBM Spectrum Scale Cc: gpfsug-discuss-bounces at spectrumscale.org ; gpfsug main discussion list ; Leo Luan Subject: Re: [gpfsug-discuss] Compression question Caution - This email from zhouhz at cn.ibm.com originated outside Imperial The statement from Olaf and Alex in below emails are correct. Firstly, compressing and decompressing files in active file system doesn't not trigger data blocks copy-on-write, that is just deallocating the unnecessary original data blocks and put compressed data in few data blocks, and no data blocks copy to snapshot. Secondly, when reading the file from snapshot, it will be redirected to active file system because there's no data blocks in snapshot, and then doing decompression in-memory for snapshot read, while the data on disk is still kept compressed. Regards, Hai Zhong Zhou -----Huzefa H Pancha/India/IBM wrote: ----- To: Hai Zhong HZ Zhou/China/IBM at IBMCN From: IBM Spectrum Scale/Poughkeepsie/IBM Sent by: Huzefa H Pancha/India/IBM Date: 11/29/2019 02:33AM Cc: gpfsug-discuss-bounces at spectrumscale.org, gpfsug main discussion list >, Leo Luan/Almaden/IBM at IBMUS Subject: Re: [EXTERNAL] Re: [gpfsug-discuss] Compression question Hai Zhong, Can you please help the customer with their compression related query. Regards, The Spectrum Scale (GPFS) team ------------------------------------------------------------------------------------------------------------------ If you feel that your question can benefit other users of Spectrum Scale (GPFS), then please post it to the public IBM developerWroks Forum at https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000000479. If your query concerns a potential software error in Spectrum Scale (GPFS) and you have an IBM software maintenance contract please contact 1-800-237-5511 in the United States or your local IBM Service Center in other countries. The forum is informally monitored as time permits and should not be used for priority messages to the Spectrum Scale (GPFS) team. [Inactive hide details for "Daniel Kidger" ---28-11-2019 20:58:12---Olaf's explanation makes excellent sense. So is this defi]"Daniel Kidger" ---28-11-2019 20:58:12---Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshot From: "Daniel Kidger" > To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Date: 28-11-2019 20:58 Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ Olaf's explanation makes excellent sense. So is this definitely the case? .. The now snapshotted inode is not updated when a live file is compressed, as it point to the current inode which itself has compressed blocks (and the compressed flag set). And likewise for deeper (older) snapshots. sDaniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Olaf Weiser" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 15:01 Hi Alex, not 100% sure about my answer.. but so far as I see it.. it is working, because of the so called "dito resolution " .. In the snaphost's inode .. die pointer to the DA's point the the next (more recent) inode information .. so accessing a file in a snapshot- "redirects" the request to the origin inode - and there ..the information about compression is given and points to the origin DA (of course.. only as long nobody changed the file since the snapshot was taken) From: "Alexander Wolf" > To: gpfsug-discuss at spectrumscale.org Date: 11/28/2019 07:03 AM Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Sent by: gpfsug-discuss-bounces at spectrumscale.org ________________________________ I see the same behavior of mmlsattr on my system (with some post 5.0.4 development build). Funny enough if I look at the file content in the snapshot it gets properly decompressed. Mit freundlichen Gr??en / Kind regards [cid:1574992361973] [IBM Spectrum Scale] Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com [cid:1574992361975] IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: "gpfsug main discussion list" > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 14:00 Which version are you running? I was involved on a big for compressed file sets and snapshots that were related to what you see. -- Cheers On 28. Nov 2019, at 14.57, Cregan, Bob > wrote: Hi Sounds logical - except the snap metadata does not have the compression flag set. So if the inode now points to a set of compressed blocks how does the client know to decompress it? After compression of an existing file we get in the snap -bash-4.2$ mmlsattr -L .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf file name: .snapshots/@GMT-2019.11.27-19.30.14/UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: @GMT-2019.11.27-19.30.14 creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE Encrypted: no and the original file is definitely compressed. -bash-4.2$ mmlsattr -L UserGuide_13.06.pdf file name: UserGuide_13.06.pdf metadata replication: 2 max 2 data replication: 1 max 3 immutable: no appendOnly: no flags: storage pool name: sata1 fileset name: userdirs snapshot name: creation time: Tue Mar 5 16:16:40 2019 Misc attributes: ARCHIVE COMPRESSION (library z) Encrypted: no Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: gpfsug-discuss-bounces at spectrumscale.org > on behalf of Daniel Kidger > Sent: 28 November 2019 12:30 To: gpfsug-discuss at spectrumscale.org > Cc: gpfsug-discuss at spectrumscale.org > Subject: Re: [gpfsug-discuss] Compression question Caution - This email from daniel.kidger at uk.ibm.com originated outside Imperial Alexander, Can you then confirm then that the inodes in the snapshot will now point to fewer but compressed blocks ? Daniel _________________________________________________________ Daniel Kidger IBM Technical Sales Specialist Spectrum Scale, Spectrum NAS and IBM Cloud Object Store +44-(0)7818 522 266 daniel.kidger at uk.ibm.com ----- Original message ----- From: "Alexander Wolf" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:21 I just tested this. Compressing a file did free up space in the file system. Looks like our compression code does not trigger COW on the snapshot. You can test this yourself by looking into mmlssnapshot -d (please not on a large production fs, this command is expensive). Mit freundlichen Gr??en / Kind regards Dr. Alexander Wolf-Reber Spectrum Scale Release Lead Architect Department M069 / Spectrum Scale Software Development +49-160-90540880 a.wolf-reber at de.ibm.com IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzende des Aufsichtsrats: Matthias Hartmann / Gesch?ftsf?hrung: Dirk Wittkopp Sitz der Gesellschaft: B?blingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 ----- Original message ----- From: "Luis Bolinches" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug-discuss at spectrumscale.org Cc: gpfsug-discuss at spectrumscale.org Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 11:47 Hi Same principle COW. The data blocks do not get modified. -- Yst?v?llisin terveisin / Kind regards / Saludos cordiales / Salutations / Salutacions Luis Bolinches Consultant IT Specialist Mobile Phone: +358503112585 https://www.youracclaim.com/user/luis-bolinches "If you always give you will always have" -- Anonymous ----- Original message ----- From: "Cregan, Bob" > Sent by: gpfsug-discuss-bounces at spectrumscale.org To: gpfsug main discussion list > Cc: Subject: [EXTERNAL] Re: [gpfsug-discuss] Compression question Date: Thu, Nov 28, 2019 12:06 PM Just to clarify this is SS compression, so mmchattr --compression yes or an ILM equivalent So not a regular modification. Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE ________________________________ From: Cregan, Bob Sent: 28 November 2019 09:43 To: gpfsug main discussion list > Subject: Compression question Hi All, Can someone answer the following question on compression in a snapshot context? We have tried various experiments and they are inconclusive - too tedious to go into the detail. What happens to the snapshot when a file is compressed in SS? The logic as I see it is ####### In a non compressed situation ############### 1) create a file, 2) create a snapshot. 3) modify a file in a normal way - the blocks on disk are changed and the old blocks are then written to the snap. ######In a compressed situation ############ 1) create a file, 2) create a snapshot. 3) Compress the file. Now IBM says the blocks are rewritten when the file is compressed. So do the old uncompressed blocks go into the snap? If so we now have 2 copies of the file and unless the compression > 50% we have used more space until the snap is deleted. You get the space back in the end, but if you are in a tight situation then potentially compression might not work for you in the short term. Thanks Bob Bob Cregan HPC Systems Analyst Information & Communication Technologies Imperial College London, South Kensington Campus London, SW7 2AZ T: 07712388129 E: b.cregan at imperial.ac.uk W: www.imperial.ac.uk/ict/rcs @imperialRCS @imperialRSE _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Ellei edell? ole toisin mainittu: / Unless stated otherwise above: Oy IBM Finland Ab PL 265, 00101 Helsinki, Finland Business ID, Y-tunnus: 0195876-3 Registered in Finland _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361973.png Type: image/png Size: 1134 bytes Desc: Image.1574992361973.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361974.png Type: image/png Size: 6645 bytes Desc: Image.1574992361974.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Image.1574992361975.png Type: image/png Size: 1134 bytes Desc: Image.1574992361975.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505984389.png Type: image/png Size: 1641 bytes Desc: Outlook-1505984389.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook-1505983882.png Type: image/png Size: 17519 bytes Desc: Outlook-1505983882.png URL: From kraemerf at de.ibm.com Fri Nov 29 08:35:22 2019 From: kraemerf at de.ibm.com (Frank Kraemer) Date: Fri, 29 Nov 2019 09:35:22 +0100 Subject: [gpfsug-discuss] FYI - Nvidia Magnum IO SC19 Accelerating data for NVIDIA GPUs with IBM Spectrum Scale Message-ID: Nvidia Magnum IO SC19 Accelerating data for NVIDIA GPUs with IBM Spectrum Scale NVIDIA announced the Magnum IO solution that complements leading-edge data management systems such as IBM Spectrum Scale and helps address AI and big data analytics I/O challenges. https://news.developer.nvidia.com/magnum-io-software-suite/ https://youtu.be/_iIAEflTZuE Accelerating data for NVIDIA GPUs - IBM IT Infrastructure Blog https://www.ibm.com/blogs/systems/accelerating-data-for-nvidia-gpus/ -frank- P.S. Oracle Cloud is now using Spectrum Scale :-) https://blogs.oracle.com/cloud-infrastructure/the-fastest-file-servers-in-the-public-cloud P.S. and the winner is ?! CM> "Coldago is a relatively little-known analyst firm and its output stands up well against prominent research firms such as Gartner and Forrester." https://blocksandfiles.com/2019/11/28/coldago-research-26-file-storage-suppliers/ Frank Kraemer IBM Consulting IT Specialist / Client Technical Architect Am Weiher 24, 65451 Kelsterbach, Germany mailto:kraemerf at de.ibm.com Mobile +49171-3043699 IBM Germany -------------- next part -------------- An HTML attachment was scrubbed... URL: