From chair at gpfsug.org  Fri May  2 15:49:54 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Fri, 02 May 2014 15:49:54 +0100
Subject: [gpfsug-discuss] GPFS UG 10 Presentations available
Message-ID: <5363B092.8020507@gpfsug.org>

Hello all

   Firstly, thanks for the feedback we've had so far.  Very much 
appreciated.

Secondly, GPFS UG 10 Presentations are now available on the 
Presentations section of the website.
Any outstanding presentations will follow shortly.

See: http://www.gpfsug.org/

Best regards,

Jez

UG Chair


From oester at gmail.com  Fri May  2 16:06:39 2014
From: oester at gmail.com (Bob Oesterlin)
Date: Fri, 2 May 2014 10:06:39 -0500
Subject: [gpfsug-discuss] GPFS UG 10 Presentations - Sven Oehme
Message-ID: <CAMNdFvAjd=oNzygAT2MDF2B1Ps6+VcALjOWu-TUywm2Q1ytd7w@mail.gmail.com>

It Sven's presentation, he mentions a tools "trcio" (in
/xcat/oehmes/gpfs-clone)

Where can I find that?

Bob Oesterlin



On Fri, May 2, 2014 at 9:49 AM, Jez Tucker (Chair) <chair at gpfsug.org> wrote:

> Hello all
>
>   Firstly, thanks for the feedback we've had so far.  Very much
> appreciated.
>
> Secondly, GPFS UG 10 Presentations are now available on the Presentations
> section of the website.
> Any outstanding presentations will follow shortly.
>
> See: http://www.gpfsug.org/
>
> Best regards,
>
> Jez
>
> UG Chair
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140502/dc657ca8/attachment.htm>

From aortiz at serviware.com  Mon May  5 10:49:53 2014
From: aortiz at serviware.com (Aurelien Ortiz  ( Serviware Toulouse ))
Date: Mon, 5 May 2014 11:49:53 +0200
Subject: [gpfsug-discuss] Hello !
Message-ID: <2F5DD237-3157-4941-B110-3F92E8ED1B4D@serviware.fr>

Hello, 

my name is Aur?lien Ortiz. 
I work at Serviware which is a subsidiary of Bull (french manufacturer). 

Serviware sells and installs clusters of small and medium size, with GPFS as the main storage solution. We use it either as a fast scratch space or a more reliable storage space in order to store results of computation.

Some of our customers : PSA, TOTAL, Renault, EDF, CNRS, CEA, IPGP, IPA, Cenaero, IFPEN, INRIA, etc.

I'm looking forward to share about GPFS with you.

Regards,
?
Aur?lien Ortiz
HPC engineer
Serviware / BULL



From luke.raimbach at oerc.ox.ac.uk  Wed May  7 10:28:59 2014
From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach)
Date: Wed, 7 May 2014 09:28:59 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <F68FAAD16AEC9744BD921F91014D2F00371189@MBX06.ad.oak.ox.ac.uk>

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639




From Paul.Sanchez at deshaw.com  Wed May  7 11:59:30 2014
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 7 May 2014 10:59:30 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <201D6001C896B846A9CFC2E841986AC11A259268@mailnycmb2a.winmail.deshaw.com>

Hi Luke,

When using RFC 1918 space among remote clusters, GPFS assumes that each cluster's privately addressed networks are not reachable from one another.  You must add explicit shared subnets via mmchconfig. Try setting subnets as follows:

gpfs.oerc.local:
subnets="10.200.0.0 10.200.0.0/cpdn.oerc.local"

cpdn.oerc.local:
subnets="10.200.0.0 10.200.0.0/gpfs.oerc.local"

I think you may also need to set the cipher list locally on each cluster to AUTHONLY via mmauth. On my clusters, these match. (No cluster says "none specified".)

Hope that helps,
Paul



Sent with Good (www.good.com)

________________________________
From: gpfsug-discuss-bounces at gpfsug.org on behalf of Luke Raimbach
Sent: Wednesday, May 07, 2014 5:28:59 AM
To: gpfsug-discuss at gpfsug.org
Subject: [gpfsug-discuss] GPFS Remote Mount Fails

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140507/f289cd64/attachment.htm>

From pavel.pokorny at datera.cz  Fri May  9 10:15:54 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Fri, 9 May 2014 17:15:54 +0800
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
Message-ID: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>

Hello,
our customer would like to deny some of the file extensions (.mp3, .avi,
...) to be stored on GPFS FS. Is there any way to deny some of the
extensions?
Right now we only find the solution to make a placement rule that directs
the files to metadataonly storage pool.

This leads me to second question. Is there any easy way to make a group of
extension (some type of templates) for placement rules? Because there is a
lot of extensions, customer would like to deny, and the placement rules
files start to grow :-(.

Thank you very much for your ideas and answers.
Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140509/b151676f/attachment.htm>

From jonathan at buzzard.me.uk  Fri May  9 14:21:58 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Fri, 09 May 2014 14:21:58 +0100
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
In-Reply-To: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
References: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
Message-ID: <1399641718.19065.22.camel@buzzard.phy.strath.ac.uk>

On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> Hello,
> our customer would like to deny some of the file extensions
> (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> some of the extensions?
> Right now we only find the solution to make a placement rule that
> directs the files to metadataonly storage pool.
> 

Create a really small storage pool and fill it in advance.

> 
> This leads me to second question. Is there any easy way to make a
> group of extension (some type of templates) for placement rules?
> Because there is a lot of extensions, customer would like to deny, and
> the placement rules files start to grow :-(.

Note if you deny extensions out right you are more likely to force users
to come up with work arounds. The simple expediency of changing the file
extension for example will lead you to a game of whack-a-mole.

Better to given them really rubbish performance and don't talk about it.
The chances are this is less of a business cost than dealing with the
outright banning, especially when there turns out to be some legitimate
business reason for having multimedia stuff, if even only occasionally.

So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
files. Performance will suck, but the users probably won't realize what
is going on, and are unlikely to come to support asking why their itunes
collection is really slow. That said if you are doing ILM with a
nearline pool holding most of the storage, it is likely to be near idle
most of the time so just force all multimedia stuff to your nearline
pool is adequate.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From pavel.pokorny at datera.cz  Mon May 12 03:46:40 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Mon, 12 May 2014 10:46:40 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
Message-ID: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>

Hello,
after discussion with Marc A Kaplan from IBM I finished with this solution.
So I am sharing it with all other user just in case you need it in future.

Example placement policy file:

/* --- Enable .avi so it will not be denied by default --- */
RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$'])

/* --- Deny all following extensions --- */
RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|\.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|\.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|\.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|\.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|\.if$|\.spiff$|\.tif*$'])

/* --- Enable every other file extensions --- */
RULE 'DEFAULT' SET POOL 'data'

Bye, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Sat, May 10, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: GPFS Placement rules - group file extension?
>       (Jonathan Buzzard)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 09 May 2014 14:21:58 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] GPFS Placement rules - group file
>         extension?
> Message-ID: <1399641718.19065.22.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> > Hello,
> > our customer would like to deny some of the file extensions
> > (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> > some of the extensions?
> > Right now we only find the solution to make a placement rule that
> > directs the files to metadataonly storage pool.
> >
>
> Create a really small storage pool and fill it in advance.
>
> >
> > This leads me to second question. Is there any easy way to make a
> > group of extension (some type of templates) for placement rules?
> > Because there is a lot of extensions, customer would like to deny, and
> > the placement rules files start to grow :-(.
>
> Note if you deny extensions out right you are more likely to force users
> to come up with work arounds. The simple expediency of changing the file
> extension for example will lead you to a game of whack-a-mole.
>
> Better to given them really rubbish performance and don't talk about it.
> The chances are this is less of a business cost than dealing with the
> outright banning, especially when there turns out to be some legitimate
> business reason for having multimedia stuff, if even only occasionally.
>
> So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
> files. Performance will suck, but the users probably won't realize what
> is going on, and are unlikely to come to support asking why their itunes
> collection is really slow. That said if you are doing ILM with a
> nearline pool holding most of the storage, it is likely to be near idle
> most of the time so just force all multimedia stuff to your nearline
> pool is adequate.
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 6
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140512/84920933/attachment.htm>

From jonathan at buzzard.me.uk  Mon May 12 12:12:19 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Mon, 12 May 2014 12:12:19 +0100
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
Message-ID: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>

On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> Hello,
> after discussion with Marc A Kaplan from IBM I finished with this
> solution. So I am sharing it with all other user just in case you need
> it in future.
> 
> 
> Example placement policy file:
> 
> 
> /* --- Enable .avi so it will not be denied by default --- */
> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$'])
> 
> 
> /* --- Deny all following extensions --- */
> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> \.if$|\.spiff$|\.tif*$'])
> 

To demonstrate why this is a wack-a-mole exercise, as a user of your
system I will just save in iPod friendly format, all my audio AAC
encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
audiobooks in .m4b format :-)

I would further note that it is rather trivial to pick a random
extension say .sdf and associate that with just about any program, most
of which examine the contents of the file and don't rely on the
extension.

Further noting if you are continuing in your wack-a-mole exercise I
could use .divx which you have conveniently not blocked.

Finally being the resourceful sort I will just embed it all in a PDF, or
word document, or powerpoint, or ...

I don't know exactly what your customer is expecting, but this sounds
like a management idea, with little understanding of how things work in
the real world.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From zgiles at gmail.com  Mon May 12 15:02:04 2014
From: zgiles at gmail.com (Zachary Giles)
Date: Mon, 12 May 2014 10:02:04 -0400
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
	<1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAMYZk=f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg@mail.gmail.com>

To further the wack-a-mole problem:
If you set those extensions to be in the system pool and assume the
system pool has no data-disks, it is still possible for GPFS 3.5 to
create a file in the system pool if one is utilizing the
small-files-in-metadata feature (whatever it's called now) and if the
file is 0-3.9K in size where it will be stored in the xattrs.
So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
if, say, you wanted to restrict .txt files, which are often that
small.



On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
<jonathan at buzzard.me.uk> wrote:
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
>> Hello,
>> after discussion with Marc A Kaplan from IBM I finished with this
>> solution. So I am sharing it with all other user just in case you need
>> it in future.
>>
>>
>> Example placement policy file:
>>
>>
>> /* --- Enable .avi so it will not be denied by default --- */
>> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$'])
>>
>>
>> /* --- Deny all following extensions --- */
>> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
>> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
>> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
>> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
>> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
>> \.if$|\.spiff$|\.tif*$'])
>>
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-- 
Zach Giles
zgiles at gmail.com


From pavel.pokorny at datera.cz  Tue May 13 15:35:25 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Tue, 13 May 2014 22:35:25 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28,
	Issue 8 (Pavel Pokorny)
Message-ID: <CAHy3nHL4t0vAC8c7PuFwsa6Wo-T3VCTNSpFc0D=-G9Su4MwrXw@mail.gmail.com>

Hello,
just to clarify. Customer need is very easy: Deny storing selected files
extension in specific filesets.
I also don't like the way to direct placement to metadata only pool. I also
know that small filse can actually be stored there.
I will appreciate more help how to make it. I don't think that expalanation
to customer that his need is stupid, is a good approach to the customer.
If there is or not .divx extension is not important, we care about GPFS
stuff and the extensions were selected by customer.
Thank you, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Tue, May 13, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Jonathan Buzzard)
>    2. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Zachary Giles)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 12 May 2014 12:12:19 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID: <1399893139.27622.11.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> > Hello,
> > after discussion with Marc A Kaplan from IBM I finished with this
> > solution. So I am sharing it with all other user just in case you need
> > it in future.
> >
> >
> > Example placement policy file:
> >
> >
> > /* --- Enable .avi so it will not be denied by default --- */
> > RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$'])
> >
> >
> > /* --- Deny all following extensions --- */
> > RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> > \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> > \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> > \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> > \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> > \.if$|\.spiff$|\.tif*$'])
> >
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 12 May 2014 10:02:04 -0400
> From: Zachary Giles <zgiles at gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID:
>         <CAMYZk=
> f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> To further the wack-a-mole problem:
> If you set those extensions to be in the system pool and assume the
> system pool has no data-disks, it is still possible for GPFS 3.5 to
> create a file in the system pool if one is utilizing the
> small-files-in-metadata feature (whatever it's called now) and if the
> file is 0-3.9K in size where it will be stored in the xattrs.
> So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
> if, say, you wanted to restrict .txt files, which are often that
> small.
>
>
>
> On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
> <jonathan at buzzard.me.uk> wrote:
> > On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> >> Hello,
> >> after discussion with Marc A Kaplan from IBM I finished with this
> >> solution. So I am sharing it with all other user just in case you need
> >> it in future.
> >>
> >>
> >> Example placement policy file:
> >>
> >>
> >> /* --- Enable .avi so it will not be denied by default --- */
> >> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$'])
> >>
> >>
> >> /* --- Deny all following extensions --- */
> >> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> >> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> >> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> >> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> >> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> >> \.if$|\.spiff$|\.tif*$'])
> >>
> >
> > To demonstrate why this is a wack-a-mole exercise, as a user of your
> > system I will just save in iPod friendly format, all my audio AAC
> > encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> > audiobooks in .m4b format :-)
> >
> > I would further note that it is rather trivial to pick a random
> > extension say .sdf and associate that with just about any program, most
> > of which examine the contents of the file and don't rely on the
> > extension.
> >
> > Further noting if you are continuing in your wack-a-mole exercise I
> > could use .divx which you have conveniently not blocked.
> >
> > Finally being the resourceful sort I will just embed it all in a PDF, or
> > word document, or powerpoint, or ...
> >
> > I don't know exactly what your customer is expecting, but this sounds
> > like a management idea, with little understanding of how things work in
> > the real world.
> >
> >
> > JAB.
> >
> > --
> > Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> > Fife, United Kingdom.
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> --
> Zach Giles
> zgiles at gmail.com
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 8
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140513/a9784129/attachment.htm>

From sabujp at gmail.com  Thu May 15 13:48:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 15 May 2014 07:48:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
Message-ID: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>

Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager
automatically reboots (and sometimes fails to automatically reboot) after
mmdelsnapshot is called, either from the filesystem manager itself or from
some other nsd/node . It didn't start happening immediately after we
updated to 17, but we never had this issue when we were at 3.5.0.11 . The
error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.
mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
mmdelsnapshot: Command failed.  Examine previous error messages to
determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup
- CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to
not reboot itself (which is actually worse), but the former does.


It also causes some havoc with CNFS file locking even after the
filesystem manager is rebooted and has come up :


May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
Address already in use


May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known


Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior
or know if it's fixed in 18?


Thanks,

Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140515/f6db747b/attachment.htm>

From sabujp at gmail.com  Mon May 19 16:16:14 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Mon, 19 May 2014 10:16:14 -0500
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/09e7cd91/attachment.htm>

From TOMP at il.ibm.com  Mon May 19 16:35:32 2014
From: TOMP at il.ibm.com (Tomer Perry)
Date: Mon, 19 May 2014 18:35:32 +0300
Subject: [gpfsug-discuss] gpfs 4.1 features?
In-Reply-To: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
References: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
Message-ID: <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702@il.ibm.com>

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm

Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/8d477962/attachment.htm>

From johnlang at us.ibm.com  Tue May 20 13:00:19 2014
From: johnlang at us.ibm.com (John Langlois)
Date: Tue, 20 May 2014 08:00:19 -0400
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
Message-ID: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>

Folks,

We're going through a naming transition and you will see some GPFS 
features now described under the codename Elastic Storage.  Don't let the 
codename confuse you ... I'll let you all know the moment we get the final 
Family and Product naming straight. We are changing the name to reflect 
the fact that IBM is significantly increasing investment in this 
technology area and we will stretch GPFS out into the Cloud with Open 
Stack object support (e.g. SWIFT object store guide coming in June).  In 
addition, GPFS will be available in the Softlayer Cloud Managed Service 
Offering catalog soon.  You will be seeing a lot of new and exciting news 
on our roadmaps.

New to GPFS 4.1 features include:
Native encryption and secure cryptography erase (data wiping) that is NIST 
Compliant and FIPS certified.
GPFS client-side IBM Elastic Storage Flash caches that increase I/O 
performance up to 6x
Active File Manager now supports parallel data transfers between sites, 
using multiple gateway nodes 
AFM now supports the GPFS protocol in addition to NFS protocol for file 
transfer
Much simpler socket based pricing - no more PVUs. 
Lots of Usability improvements that include
  a.  NFS Data Migration - Eases data transfer from NFS when upgrading 
hardware or buying a new system 
  b   AFM Optimization  - Improved prefetch performance; support for the 
native GPFS protocol   (described above)
  c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool; 
Backup utility enhancements
  d.  FPO Improvements - Data locality aware file system recovery and 
improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a 
Cinder driver that gives architects access to GPFS features and 
capabilities.

Here's the current external website landing page:  
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager

Phone: 1-919-267-6767
E-mail: johnlang at us.ibm.com
Find me on:   and within IBM on:  






From:   gpfsug-discuss-request at gpfsug.org
To:     gpfsug-discuss at gpfsug.org
Date:   05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                 gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                 http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                 gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                 gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. gpfs 4.1 features? (Sabuj Pattanayek)
   2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0001.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0002.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment.gif>

From sjhoward at iu.edu  Tue May 20 15:17:32 2014
From: sjhoward at iu.edu (Howard, Stewart Jameson)
Date: Tue, 20 May 2014 14:17:32 +0000
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
Message-ID: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>

Hi All,

My name is Stewart Howard and I work for Indiana University as an admin on a two-site replicated GPFS cluster.  I'm a new member of this mailing list and this is my first post  :)

Recently, we've discovered that small-file performance on our system is pretty lack-luster.  For comparison, here are some numbers:

1)  When transferring large files (~2 GB), we get outstanding performance and can typically saturate the client's network connection.  We generally see about 490 MB/s over a 10Gb line, which should be about right, given that we lose half of our bandwidth to replication.

2)  When transferring a large number of small files, we get a very poor transfer rate, generally on the order of 2 MB/s, writing from a client node *inside* the GPFS cluster.

I'm wondering if anyone else has experience with similar performance issues and what ended up being the cause/solution.  Also, I would be interested in hearing any general rules-of-thumb that the group has found helpful in balancing performance between large-file and small-file I/O.

We have gathered some diagnostic information while performing various small-file I/O operations, as well as a variety of metadata operations in quick succession.  I'd be happy to share results of the diagnostics, if it would help provide context.

Thank you so much for all of your help!

Stewart Howard
Indiana University


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/63b40754/attachment.htm>

From dhildeb at us.ibm.com  Tue May 20 18:29:51 2014
From: dhildeb at us.ibm.com (Dean Hildebrand)
Date: Tue, 20 May 2014 10:29:51 -0700
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
	<OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
Message-ID: <OF2F748B64.B172A915-ON88257CDE.00601714-88257CDE.00601DF1@us.ibm.com>


Nice email!

Dean Hildebrand
IBM Master Inventor and Manager | Scale-out Storage Software
IBM Almaden Research Center




From:	John Langlois/Raleigh/IBM at IBMUS
To:	gpfsug-discuss at gpfsug.org
Date:	05/20/2014 05:01 AM
Subject:	[gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
Sent by:	gpfsug-discuss-bounces at gpfsug.org



Folks,

We're going through a naming transition and you will see some GPFS features
now described under the codename Elastic Storage.  Don't let the codename
confuse you ... I'll let you all know the moment we get the final Family
and Product naming straight. We are changing the name to reflect the fact
that IBM is significantly increasing investment in this technology area and
we will stretch GPFS out into the Cloud with Open Stack object support
(e.g. SWIFT object store guide coming in June).  In addition, GPFS will be
available in the Softlayer Cloud Managed Service Offering catalog soon.
You will be seeing a lot of new and exciting news on our roadmaps.

New to GPFS 4.1 features include:
   1.	Native encryption and secure cryptography erase (data wiping) that is
      NIST Compliant and FIPS certified.
   2.	GPFS client-side IBM Elastic Storage Flash caches that increase I/O
      performance up to 6x
   3.	Active File Manager now supports parallel data transfers between
      sites, using multiple gateway nodes
   4.	AFM now supports the GPFS protocol in addition to NFS protocol for
      file transfer
   5.	Much simpler socket based pricing - no more PVUs.
   6.	Lots of Usability improvements that include
       a.  NFS Data Migration - Eases data transfer from NFS when upgrading
      hardware or buying a new system
       b   AFM Optimization  - Improved prefetch performance; support for
      the native GPFS protocol   (described above)
       c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool;
      Backup utility enhancements
       d.  FPO Improvements - Data locality aware file system recovery and
      improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a
Cinder driver that gives architects access to GPFS features and
capabilities.

Here's the current external website landing page:
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager
                                                                                                                                      
                                                                                                                                      
                                                                                                                                      
 Phone: 1-919-267-6767                                                                                                            IBM 
 E-mail: johnlang at us.ibm.com                                                                                                          
 Find me on: LinkedIn:                                                                                                                
 http://www.linkedin.com/profile/view?id=24557762&authType=NAME_SEARCH&authToken=b9zY&locale=en_US&sr                                 
 Twitter: https://twitter.com/johnlang123 and within IBM on: IBM Connections:                                                         
 https://w3-connections.ibm.com/profiles/html/profileView.do?userid=69bf0ec0-8f0a-1028-94ba-db07163b5                                 
                                                                                                                                      
                                                                                                                                      






From:        gpfsug-discuss-request at gpfsug.org
To:        gpfsug-discuss at gpfsug.org
Date:        05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

  1. gpfs 4.1 features? (Sabuj Pattanayek)
  2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62923814.jpg
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62235135.jpg
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62861266.jpg
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62781961.gif
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0001.gif>

From chekh at stanford.edu  Wed May 21 21:04:52 2014
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 21 May 2014 13:04:52 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
Message-ID: <537D06E4.3090603@stanford.edu>

Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860


From oehmes at us.ibm.com  Wed May 21 22:42:51 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Wed, 21 May 2014 14:42:51 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <537D06E4.3090603@stanford.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
	<537D06E4.3090603@stanford.edu>
Message-ID: <OF11EC8E07.09F9EDEE-ON88257CDF.0074EF02-88257CDF.007747E9@us.ibm.com>

Hi Alex, Stewart,

the problem is more likely a latency and lack of parallelism of the 
transfer.
 
if you have 1 thread that transfers files of 1k in size you don't get a 
lot of bandwidth as it transfers one at a time and the latency kills the 
bandwidth. to explain this on an example :

assume your network between client and server is 10 gigabit and both nodes 
are capable of pushing this.

if it takes 1 ms for reading + writing each 1 kb file you get ~ 1.02 
MB/sec
if your filesize changes to 4k, even you don't do anything else it goes up 
to 4.06 MB/sec
if you can reduce latency on read or write to lets say 100us the same 
process would transfer 37.86 MB/sec

so this is a latency / parallelism problem and this has nothing to do with 
GPFS, you would have exactly the same issue.
the solution is to copy the data with a tool that does it multi threaded 
as the numbers above are based on 1 thread. 
if you would have 100us read+write time and transfer 4k files with 10 
threads in parallel the same transfer would be close to 400 MB/sec.

hope that helps. Sven 
------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
Phone: +1 (408) 824-8904 
IBM Almaden Research Lab 
------------------------------------------



From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org
Date:   05/21/2014 01:05 PM
Subject:        Re: [gpfsug-discuss] Mitigating Poor Small-file I/O 
Performance
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140521/08d101d4/attachment.htm>

From sabujp at gmail.com  Tue May 27 22:29:02 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Tue, 27 May 2014 16:29:02 -0500
Subject: [gpfsug-discuss] feature request : ability to run multiple snapshot
	related commands simultaneously
Message-ID: <CAEeMGHuFnKtH1a8eirKqu5r5wxeOfC_wVahpwSnPio8k8XV51w@mail.gmail.com>

Sometimes snapshot deletions can take a very long time. While they're
running no other snapshot related commands can be run, e.g. it doesn't look
like I can run an mmcrsnapshot on the same fileset while a global snapshot
deletion or a snapshot deletion on the same fileset is ongoing.

Today's snapshot deletion on a global snapshot created about 19 hours ago
is going painfully slowly for some reason. It's also affecting CNFS and SMB
performance with periods of hangs for about 30+ seconds. This is not the
norm on our fs. GPFS access continues to be ok during these NFS/SMB hangs.
Another problem it causes is that snapshots on the root fileset cannot be
created while this global snapshot is being deleted. This disrupts our
daily snapshot creation schedule and I also have to switch to running
mmbackup without a snapshot, since I won't be able to create a new global
snapshot if another global snapshot is being deleted.

If possible I'd request that running multiple snapshot related commands
simultaneously be allowed, at least the ability to run an mmcrsnapshot
while an mmdelsnapshot is running would be nice. I can see that running
multiple global snapshot deletions or multiple simultaneous snapshot
deletions on the same fileset would be risky/very difficult (or impossible)
to implement but it seems like it should be possible to "quiesce" an
mmdelsnapshot for a few seconds so that an mmcrsnapshot on the same fileset
can be completed. This can be done manually by interrupting an ongoing
snapshot deletion, creating the new snapshot, and continuing the deletion
but would be nice if this could be fully automated . What would be even
cooler would be a snapshot command queuing system for deletion requests
within the same fileset, that way snapshot deletions would be queued up and
automatically occur as soon as deletions in the same fileset are complete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140527/edbbdf15/attachment.htm>

From sabujp at gmail.com  Fri May 30 02:34:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 29 May 2014 20:34:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
Message-ID: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>

This is still happening in 3.5.0.18 and when a snapshot is being deleted it
slows NFS read speeds to a crawl (but not gpfs and not NFS writes).


On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

> Hi all,
>
> We're running 3.5.0.17 now and it looks like the filesystem manager
> automatically reboots (and sometimes fails to automatically reboot) after
> mmdelsnapshot is called, either from the filesystem manager itself or from
> some other nsd/node . It didn't start happening immediately after we
> updated to 17, but we never had this issue when we were at 3.5.0.11 . The
> error mmdelsnapshot throws at some point is :
>
> Lost connection to file system daemon.
> mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
> mmdelsnapshot: Command failed.  Examine previous error messages to
> determine cause.
>
> It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.
>
>
> It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :
>
>
> May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
> Address already in use
>
>
> May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port
>
> for RPC socket: Name or service not known
>
>
> Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?
>
>
> Thanks,
>
> Sabuj
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140529/76d1258b/attachment.htm>

From bbanister at jumptrading.com  Fri May 30 14:35:55 2014
From: bbanister at jumptrading.com (Bryan Banister)
Date: Fri, 30 May 2014 13:35:55 +0000
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>

This sounds like a serious problem and you should open a PMR with IBM to get direct guidance.

I normally will take a GPFS trace during a problem like this from all of the nodes that are affected or directly involved in the operation.

Hope that helps,
-Bryan

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Sabuj Pattanayek
Sent: Thursday, May 29, 2014 8:34 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] filesystem manager crashes every time mmdelsnapshot (from either the filesystem manager or some other nsd/client) is called

This is still happening in 3.5.0.18 and when a snapshot is being deleted it slows NFS read speeds to a crawl (but not gpfs and not NFS writes).

On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com<mailto:sabujp at gmail.com>> wrote:
Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager automatically reboots (and sometimes fails to automatically reboot) after mmdelsnapshot is called, either from the filesystem manager itself or from some other nsd/node . It didn't start happening immediately after we updated to 17, but we never had this issue when we were at 3.5.0.11 . The error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.

mmdelsnapshot: An internode connection between GPFS nodes was disrupted.

mmdelsnapshot: Command failed.  Examine previous error messages to

determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.



It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :



May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:

Address already in use



May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known



Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?



Thanks,

Sabuj



________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/d06dcb69/attachment.htm>

From sabujp at gmail.com  Fri May 30 14:49:15 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 30 May 2014 08:49:15 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
	<21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <CAEeMGHuGfMrVPYd97wPQ_FfCj=ocEzFb7fP9Rx0zQC623==b=A@mail.gmail.com>

On Fri, May 30, 2014 at 8:35 AM, Bryan Banister <bbanister at jumptrading.com>
wrote:

>  This sounds like a serious problem and you should open a PMR with IBM to
> get direct guidance.
>
>
>
> I normally will take a GPFS trace during a problem like this from all of
> the nodes that are affected or directly involved in the operation.
>

easier said than done, I've tried at least 10 times. It's almost as if the
act of running the mmtrace prevents the error from occurring, we've sent
internaldumps and gpfs.snap that were generated to ddn. Not having much
luck in figuring out if it's been sent through to IBM yet or not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/dd6729bd/attachment.htm>

From chair at gpfsug.org  Fri May  2 15:49:54 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Fri, 02 May 2014 15:49:54 +0100
Subject: [gpfsug-discuss] GPFS UG 10 Presentations available
Message-ID: <5363B092.8020507@gpfsug.org>

Hello all

   Firstly, thanks for the feedback we've had so far.  Very much 
appreciated.

Secondly, GPFS UG 10 Presentations are now available on the 
Presentations section of the website.
Any outstanding presentations will follow shortly.

See: http://www.gpfsug.org/

Best regards,

Jez

UG Chair


From oester at gmail.com  Fri May  2 16:06:39 2014
From: oester at gmail.com (Bob Oesterlin)
Date: Fri, 2 May 2014 10:06:39 -0500
Subject: [gpfsug-discuss] GPFS UG 10 Presentations - Sven Oehme
Message-ID: <CAMNdFvAjd=oNzygAT2MDF2B1Ps6+VcALjOWu-TUywm2Q1ytd7w@mail.gmail.com>

It Sven's presentation, he mentions a tools "trcio" (in
/xcat/oehmes/gpfs-clone)

Where can I find that?

Bob Oesterlin



On Fri, May 2, 2014 at 9:49 AM, Jez Tucker (Chair) <chair at gpfsug.org> wrote:

> Hello all
>
>   Firstly, thanks for the feedback we've had so far.  Very much
> appreciated.
>
> Secondly, GPFS UG 10 Presentations are now available on the Presentations
> section of the website.
> Any outstanding presentations will follow shortly.
>
> See: http://www.gpfsug.org/
>
> Best regards,
>
> Jez
>
> UG Chair
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140502/dc657ca8/attachment-0001.htm>

From aortiz at serviware.com  Mon May  5 10:49:53 2014
From: aortiz at serviware.com (Aurelien Ortiz  ( Serviware Toulouse ))
Date: Mon, 5 May 2014 11:49:53 +0200
Subject: [gpfsug-discuss] Hello !
Message-ID: <2F5DD237-3157-4941-B110-3F92E8ED1B4D@serviware.fr>

Hello, 

my name is Aur?lien Ortiz. 
I work at Serviware which is a subsidiary of Bull (french manufacturer). 

Serviware sells and installs clusters of small and medium size, with GPFS as the main storage solution. We use it either as a fast scratch space or a more reliable storage space in order to store results of computation.

Some of our customers : PSA, TOTAL, Renault, EDF, CNRS, CEA, IPGP, IPA, Cenaero, IFPEN, INRIA, etc.

I'm looking forward to share about GPFS with you.

Regards,
?
Aur?lien Ortiz
HPC engineer
Serviware / BULL



From luke.raimbach at oerc.ox.ac.uk  Wed May  7 10:28:59 2014
From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach)
Date: Wed, 7 May 2014 09:28:59 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <F68FAAD16AEC9744BD921F91014D2F00371189@MBX06.ad.oak.ox.ac.uk>

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639




From Paul.Sanchez at deshaw.com  Wed May  7 11:59:30 2014
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 7 May 2014 10:59:30 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <201D6001C896B846A9CFC2E841986AC11A259268@mailnycmb2a.winmail.deshaw.com>

Hi Luke,

When using RFC 1918 space among remote clusters, GPFS assumes that each cluster's privately addressed networks are not reachable from one another.  You must add explicit shared subnets via mmchconfig. Try setting subnets as follows:

gpfs.oerc.local:
subnets="10.200.0.0 10.200.0.0/cpdn.oerc.local"

cpdn.oerc.local:
subnets="10.200.0.0 10.200.0.0/gpfs.oerc.local"

I think you may also need to set the cipher list locally on each cluster to AUTHONLY via mmauth. On my clusters, these match. (No cluster says "none specified".)

Hope that helps,
Paul



Sent with Good (www.good.com)

________________________________
From: gpfsug-discuss-bounces at gpfsug.org on behalf of Luke Raimbach
Sent: Wednesday, May 07, 2014 5:28:59 AM
To: gpfsug-discuss at gpfsug.org
Subject: [gpfsug-discuss] GPFS Remote Mount Fails

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140507/f289cd64/attachment-0001.htm>

From pavel.pokorny at datera.cz  Fri May  9 10:15:54 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Fri, 9 May 2014 17:15:54 +0800
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
Message-ID: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>

Hello,
our customer would like to deny some of the file extensions (.mp3, .avi,
...) to be stored on GPFS FS. Is there any way to deny some of the
extensions?
Right now we only find the solution to make a placement rule that directs
the files to metadataonly storage pool.

This leads me to second question. Is there any easy way to make a group of
extension (some type of templates) for placement rules? Because there is a
lot of extensions, customer would like to deny, and the placement rules
files start to grow :-(.

Thank you very much for your ideas and answers.
Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140509/b151676f/attachment-0001.htm>

From jonathan at buzzard.me.uk  Fri May  9 14:21:58 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Fri, 09 May 2014 14:21:58 +0100
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
In-Reply-To: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
References: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
Message-ID: <1399641718.19065.22.camel@buzzard.phy.strath.ac.uk>

On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> Hello,
> our customer would like to deny some of the file extensions
> (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> some of the extensions?
> Right now we only find the solution to make a placement rule that
> directs the files to metadataonly storage pool.
> 

Create a really small storage pool and fill it in advance.

> 
> This leads me to second question. Is there any easy way to make a
> group of extension (some type of templates) for placement rules?
> Because there is a lot of extensions, customer would like to deny, and
> the placement rules files start to grow :-(.

Note if you deny extensions out right you are more likely to force users
to come up with work arounds. The simple expediency of changing the file
extension for example will lead you to a game of whack-a-mole.

Better to given them really rubbish performance and don't talk about it.
The chances are this is less of a business cost than dealing with the
outright banning, especially when there turns out to be some legitimate
business reason for having multimedia stuff, if even only occasionally.

So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
files. Performance will suck, but the users probably won't realize what
is going on, and are unlikely to come to support asking why their itunes
collection is really slow. That said if you are doing ILM with a
nearline pool holding most of the storage, it is likely to be near idle
most of the time so just force all multimedia stuff to your nearline
pool is adequate.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From pavel.pokorny at datera.cz  Mon May 12 03:46:40 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Mon, 12 May 2014 10:46:40 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
Message-ID: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>

Hello,
after discussion with Marc A Kaplan from IBM I finished with this solution.
So I am sharing it with all other user just in case you need it in future.

Example placement policy file:

/* --- Enable .avi so it will not be denied by default --- */
RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$'])

/* --- Deny all following extensions --- */
RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|\.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|\.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|\.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|\.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|\.if$|\.spiff$|\.tif*$'])

/* --- Enable every other file extensions --- */
RULE 'DEFAULT' SET POOL 'data'

Bye, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Sat, May 10, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: GPFS Placement rules - group file extension?
>       (Jonathan Buzzard)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 09 May 2014 14:21:58 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] GPFS Placement rules - group file
>         extension?
> Message-ID: <1399641718.19065.22.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> > Hello,
> > our customer would like to deny some of the file extensions
> > (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> > some of the extensions?
> > Right now we only find the solution to make a placement rule that
> > directs the files to metadataonly storage pool.
> >
>
> Create a really small storage pool and fill it in advance.
>
> >
> > This leads me to second question. Is there any easy way to make a
> > group of extension (some type of templates) for placement rules?
> > Because there is a lot of extensions, customer would like to deny, and
> > the placement rules files start to grow :-(.
>
> Note if you deny extensions out right you are more likely to force users
> to come up with work arounds. The simple expediency of changing the file
> extension for example will lead you to a game of whack-a-mole.
>
> Better to given them really rubbish performance and don't talk about it.
> The chances are this is less of a business cost than dealing with the
> outright banning, especially when there turns out to be some legitimate
> business reason for having multimedia stuff, if even only occasionally.
>
> So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
> files. Performance will suck, but the users probably won't realize what
> is going on, and are unlikely to come to support asking why their itunes
> collection is really slow. That said if you are doing ILM with a
> nearline pool holding most of the storage, it is likely to be near idle
> most of the time so just force all multimedia stuff to your nearline
> pool is adequate.
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 6
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140512/84920933/attachment-0001.htm>

From jonathan at buzzard.me.uk  Mon May 12 12:12:19 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Mon, 12 May 2014 12:12:19 +0100
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
Message-ID: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>

On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> Hello,
> after discussion with Marc A Kaplan from IBM I finished with this
> solution. So I am sharing it with all other user just in case you need
> it in future.
> 
> 
> Example placement policy file:
> 
> 
> /* --- Enable .avi so it will not be denied by default --- */
> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$'])
> 
> 
> /* --- Deny all following extensions --- */
> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> \.if$|\.spiff$|\.tif*$'])
> 

To demonstrate why this is a wack-a-mole exercise, as a user of your
system I will just save in iPod friendly format, all my audio AAC
encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
audiobooks in .m4b format :-)

I would further note that it is rather trivial to pick a random
extension say .sdf and associate that with just about any program, most
of which examine the contents of the file and don't rely on the
extension.

Further noting if you are continuing in your wack-a-mole exercise I
could use .divx which you have conveniently not blocked.

Finally being the resourceful sort I will just embed it all in a PDF, or
word document, or powerpoint, or ...

I don't know exactly what your customer is expecting, but this sounds
like a management idea, with little understanding of how things work in
the real world.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From zgiles at gmail.com  Mon May 12 15:02:04 2014
From: zgiles at gmail.com (Zachary Giles)
Date: Mon, 12 May 2014 10:02:04 -0400
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
	<1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAMYZk=f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg@mail.gmail.com>

To further the wack-a-mole problem:
If you set those extensions to be in the system pool and assume the
system pool has no data-disks, it is still possible for GPFS 3.5 to
create a file in the system pool if one is utilizing the
small-files-in-metadata feature (whatever it's called now) and if the
file is 0-3.9K in size where it will be stored in the xattrs.
So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
if, say, you wanted to restrict .txt files, which are often that
small.



On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
<jonathan at buzzard.me.uk> wrote:
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
>> Hello,
>> after discussion with Marc A Kaplan from IBM I finished with this
>> solution. So I am sharing it with all other user just in case you need
>> it in future.
>>
>>
>> Example placement policy file:
>>
>>
>> /* --- Enable .avi so it will not be denied by default --- */
>> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$'])
>>
>>
>> /* --- Deny all following extensions --- */
>> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
>> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
>> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
>> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
>> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
>> \.if$|\.spiff$|\.tif*$'])
>>
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-- 
Zach Giles
zgiles at gmail.com


From pavel.pokorny at datera.cz  Tue May 13 15:35:25 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Tue, 13 May 2014 22:35:25 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28,
	Issue 8 (Pavel Pokorny)
Message-ID: <CAHy3nHL4t0vAC8c7PuFwsa6Wo-T3VCTNSpFc0D=-G9Su4MwrXw@mail.gmail.com>

Hello,
just to clarify. Customer need is very easy: Deny storing selected files
extension in specific filesets.
I also don't like the way to direct placement to metadata only pool. I also
know that small filse can actually be stored there.
I will appreciate more help how to make it. I don't think that expalanation
to customer that his need is stupid, is a good approach to the customer.
If there is or not .divx extension is not important, we care about GPFS
stuff and the extensions were selected by customer.
Thank you, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Tue, May 13, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Jonathan Buzzard)
>    2. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Zachary Giles)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 12 May 2014 12:12:19 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID: <1399893139.27622.11.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> > Hello,
> > after discussion with Marc A Kaplan from IBM I finished with this
> > solution. So I am sharing it with all other user just in case you need
> > it in future.
> >
> >
> > Example placement policy file:
> >
> >
> > /* --- Enable .avi so it will not be denied by default --- */
> > RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$'])
> >
> >
> > /* --- Deny all following extensions --- */
> > RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> > \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> > \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> > \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> > \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> > \.if$|\.spiff$|\.tif*$'])
> >
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 12 May 2014 10:02:04 -0400
> From: Zachary Giles <zgiles at gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID:
>         <CAMYZk=
> f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> To further the wack-a-mole problem:
> If you set those extensions to be in the system pool and assume the
> system pool has no data-disks, it is still possible for GPFS 3.5 to
> create a file in the system pool if one is utilizing the
> small-files-in-metadata feature (whatever it's called now) and if the
> file is 0-3.9K in size where it will be stored in the xattrs.
> So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
> if, say, you wanted to restrict .txt files, which are often that
> small.
>
>
>
> On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
> <jonathan at buzzard.me.uk> wrote:
> > On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> >> Hello,
> >> after discussion with Marc A Kaplan from IBM I finished with this
> >> solution. So I am sharing it with all other user just in case you need
> >> it in future.
> >>
> >>
> >> Example placement policy file:
> >>
> >>
> >> /* --- Enable .avi so it will not be denied by default --- */
> >> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$'])
> >>
> >>
> >> /* --- Deny all following extensions --- */
> >> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> >> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> >> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> >> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> >> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> >> \.if$|\.spiff$|\.tif*$'])
> >>
> >
> > To demonstrate why this is a wack-a-mole exercise, as a user of your
> > system I will just save in iPod friendly format, all my audio AAC
> > encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> > audiobooks in .m4b format :-)
> >
> > I would further note that it is rather trivial to pick a random
> > extension say .sdf and associate that with just about any program, most
> > of which examine the contents of the file and don't rely on the
> > extension.
> >
> > Further noting if you are continuing in your wack-a-mole exercise I
> > could use .divx which you have conveniently not blocked.
> >
> > Finally being the resourceful sort I will just embed it all in a PDF, or
> > word document, or powerpoint, or ...
> >
> > I don't know exactly what your customer is expecting, but this sounds
> > like a management idea, with little understanding of how things work in
> > the real world.
> >
> >
> > JAB.
> >
> > --
> > Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> > Fife, United Kingdom.
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> --
> Zach Giles
> zgiles at gmail.com
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 8
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140513/a9784129/attachment-0001.htm>

From sabujp at gmail.com  Thu May 15 13:48:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 15 May 2014 07:48:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
Message-ID: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>

Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager
automatically reboots (and sometimes fails to automatically reboot) after
mmdelsnapshot is called, either from the filesystem manager itself or from
some other nsd/node . It didn't start happening immediately after we
updated to 17, but we never had this issue when we were at 3.5.0.11 . The
error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.
mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
mmdelsnapshot: Command failed.  Examine previous error messages to
determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup
- CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to
not reboot itself (which is actually worse), but the former does.


It also causes some havoc with CNFS file locking even after the
filesystem manager is rebooted and has come up :


May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
Address already in use


May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known


Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior
or know if it's fixed in 18?


Thanks,

Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140515/f6db747b/attachment-0001.htm>

From sabujp at gmail.com  Mon May 19 16:16:14 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Mon, 19 May 2014 10:16:14 -0500
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/09e7cd91/attachment-0001.htm>

From TOMP at il.ibm.com  Mon May 19 16:35:32 2014
From: TOMP at il.ibm.com (Tomer Perry)
Date: Mon, 19 May 2014 18:35:32 +0300
Subject: [gpfsug-discuss] gpfs 4.1 features?
In-Reply-To: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
References: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
Message-ID: <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702@il.ibm.com>

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm

Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/8d477962/attachment-0001.htm>

From johnlang at us.ibm.com  Tue May 20 13:00:19 2014
From: johnlang at us.ibm.com (John Langlois)
Date: Tue, 20 May 2014 08:00:19 -0400
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
Message-ID: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>

Folks,

We're going through a naming transition and you will see some GPFS 
features now described under the codename Elastic Storage.  Don't let the 
codename confuse you ... I'll let you all know the moment we get the final 
Family and Product naming straight. We are changing the name to reflect 
the fact that IBM is significantly increasing investment in this 
technology area and we will stretch GPFS out into the Cloud with Open 
Stack object support (e.g. SWIFT object store guide coming in June).  In 
addition, GPFS will be available in the Softlayer Cloud Managed Service 
Offering catalog soon.  You will be seeing a lot of new and exciting news 
on our roadmaps.

New to GPFS 4.1 features include:
Native encryption and secure cryptography erase (data wiping) that is NIST 
Compliant and FIPS certified.
GPFS client-side IBM Elastic Storage Flash caches that increase I/O 
performance up to 6x
Active File Manager now supports parallel data transfers between sites, 
using multiple gateway nodes 
AFM now supports the GPFS protocol in addition to NFS protocol for file 
transfer
Much simpler socket based pricing - no more PVUs. 
Lots of Usability improvements that include
  a.  NFS Data Migration - Eases data transfer from NFS when upgrading 
hardware or buying a new system 
  b   AFM Optimization  - Improved prefetch performance; support for the 
native GPFS protocol   (described above)
  c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool; 
Backup utility enhancements
  d.  FPO Improvements - Data locality aware file system recovery and 
improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a 
Cinder driver that gives architects access to GPFS features and 
capabilities.

Here's the current external website landing page:  
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager

Phone: 1-919-267-6767
E-mail: johnlang at us.ibm.com
Find me on:   and within IBM on:  






From:   gpfsug-discuss-request at gpfsug.org
To:     gpfsug-discuss at gpfsug.org
Date:   05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                 gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                 http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                 gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                 gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. gpfs 4.1 features? (Sabuj Pattanayek)
   2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0003.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0004.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0005.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0001.gif>

From sjhoward at iu.edu  Tue May 20 15:17:32 2014
From: sjhoward at iu.edu (Howard, Stewart Jameson)
Date: Tue, 20 May 2014 14:17:32 +0000
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
Message-ID: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>

Hi All,

My name is Stewart Howard and I work for Indiana University as an admin on a two-site replicated GPFS cluster.  I'm a new member of this mailing list and this is my first post  :)

Recently, we've discovered that small-file performance on our system is pretty lack-luster.  For comparison, here are some numbers:

1)  When transferring large files (~2 GB), we get outstanding performance and can typically saturate the client's network connection.  We generally see about 490 MB/s over a 10Gb line, which should be about right, given that we lose half of our bandwidth to replication.

2)  When transferring a large number of small files, we get a very poor transfer rate, generally on the order of 2 MB/s, writing from a client node *inside* the GPFS cluster.

I'm wondering if anyone else has experience with similar performance issues and what ended up being the cause/solution.  Also, I would be interested in hearing any general rules-of-thumb that the group has found helpful in balancing performance between large-file and small-file I/O.

We have gathered some diagnostic information while performing various small-file I/O operations, as well as a variety of metadata operations in quick succession.  I'd be happy to share results of the diagnostics, if it would help provide context.

Thank you so much for all of your help!

Stewart Howard
Indiana University


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/63b40754/attachment-0001.htm>

From dhildeb at us.ibm.com  Tue May 20 18:29:51 2014
From: dhildeb at us.ibm.com (Dean Hildebrand)
Date: Tue, 20 May 2014 10:29:51 -0700
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
	<OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
Message-ID: <OF2F748B64.B172A915-ON88257CDE.00601714-88257CDE.00601DF1@us.ibm.com>


Nice email!

Dean Hildebrand
IBM Master Inventor and Manager | Scale-out Storage Software
IBM Almaden Research Center




From:	John Langlois/Raleigh/IBM at IBMUS
To:	gpfsug-discuss at gpfsug.org
Date:	05/20/2014 05:01 AM
Subject:	[gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
Sent by:	gpfsug-discuss-bounces at gpfsug.org



Folks,

We're going through a naming transition and you will see some GPFS features
now described under the codename Elastic Storage.  Don't let the codename
confuse you ... I'll let you all know the moment we get the final Family
and Product naming straight. We are changing the name to reflect the fact
that IBM is significantly increasing investment in this technology area and
we will stretch GPFS out into the Cloud with Open Stack object support
(e.g. SWIFT object store guide coming in June).  In addition, GPFS will be
available in the Softlayer Cloud Managed Service Offering catalog soon.
You will be seeing a lot of new and exciting news on our roadmaps.

New to GPFS 4.1 features include:
   1.	Native encryption and secure cryptography erase (data wiping) that is
      NIST Compliant and FIPS certified.
   2.	GPFS client-side IBM Elastic Storage Flash caches that increase I/O
      performance up to 6x
   3.	Active File Manager now supports parallel data transfers between
      sites, using multiple gateway nodes
   4.	AFM now supports the GPFS protocol in addition to NFS protocol for
      file transfer
   5.	Much simpler socket based pricing - no more PVUs.
   6.	Lots of Usability improvements that include
       a.  NFS Data Migration - Eases data transfer from NFS when upgrading
      hardware or buying a new system
       b   AFM Optimization  - Improved prefetch performance; support for
      the native GPFS protocol   (described above)
       c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool;
      Backup utility enhancements
       d.  FPO Improvements - Data locality aware file system recovery and
      improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a
Cinder driver that gives architects access to GPFS features and
capabilities.

Here's the current external website landing page:
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager
                                                                                                                                      
                                                                                                                                      
                                                                                                                                      
 Phone: 1-919-267-6767                                                                                                            IBM 
 E-mail: johnlang at us.ibm.com                                                                                                          
 Find me on: LinkedIn:                                                                                                                
 http://www.linkedin.com/profile/view?id=24557762&authType=NAME_SEARCH&authToken=b9zY&locale=en_US&sr                                 
 Twitter: https://twitter.com/johnlang123 and within IBM on: IBM Connections:                                                         
 https://w3-connections.ibm.com/profiles/html/profileView.do?userid=69bf0ec0-8f0a-1028-94ba-db07163b5                                 
                                                                                                                                      
                                                                                                                                      






From:        gpfsug-discuss-request at gpfsug.org
To:        gpfsug-discuss at gpfsug.org
Date:        05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

  1. gpfs 4.1 features? (Sabuj Pattanayek)
  2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62923814.jpg
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62235135.jpg
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62861266.jpg
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0005.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62781961.gif
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0003.gif>

From chekh at stanford.edu  Wed May 21 21:04:52 2014
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 21 May 2014 13:04:52 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
Message-ID: <537D06E4.3090603@stanford.edu>

Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860


From oehmes at us.ibm.com  Wed May 21 22:42:51 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Wed, 21 May 2014 14:42:51 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <537D06E4.3090603@stanford.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
	<537D06E4.3090603@stanford.edu>
Message-ID: <OF11EC8E07.09F9EDEE-ON88257CDF.0074EF02-88257CDF.007747E9@us.ibm.com>

Hi Alex, Stewart,

the problem is more likely a latency and lack of parallelism of the 
transfer.
 
if you have 1 thread that transfers files of 1k in size you don't get a 
lot of bandwidth as it transfers one at a time and the latency kills the 
bandwidth. to explain this on an example :

assume your network between client and server is 10 gigabit and both nodes 
are capable of pushing this.

if it takes 1 ms for reading + writing each 1 kb file you get ~ 1.02 
MB/sec
if your filesize changes to 4k, even you don't do anything else it goes up 
to 4.06 MB/sec
if you can reduce latency on read or write to lets say 100us the same 
process would transfer 37.86 MB/sec

so this is a latency / parallelism problem and this has nothing to do with 
GPFS, you would have exactly the same issue.
the solution is to copy the data with a tool that does it multi threaded 
as the numbers above are based on 1 thread. 
if you would have 100us read+write time and transfer 4k files with 10 
threads in parallel the same transfer would be close to 400 MB/sec.

hope that helps. Sven 
------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
Phone: +1 (408) 824-8904 
IBM Almaden Research Lab 
------------------------------------------



From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org
Date:   05/21/2014 01:05 PM
Subject:        Re: [gpfsug-discuss] Mitigating Poor Small-file I/O 
Performance
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140521/08d101d4/attachment-0001.htm>

From sabujp at gmail.com  Tue May 27 22:29:02 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Tue, 27 May 2014 16:29:02 -0500
Subject: [gpfsug-discuss] feature request : ability to run multiple snapshot
	related commands simultaneously
Message-ID: <CAEeMGHuFnKtH1a8eirKqu5r5wxeOfC_wVahpwSnPio8k8XV51w@mail.gmail.com>

Sometimes snapshot deletions can take a very long time. While they're
running no other snapshot related commands can be run, e.g. it doesn't look
like I can run an mmcrsnapshot on the same fileset while a global snapshot
deletion or a snapshot deletion on the same fileset is ongoing.

Today's snapshot deletion on a global snapshot created about 19 hours ago
is going painfully slowly for some reason. It's also affecting CNFS and SMB
performance with periods of hangs for about 30+ seconds. This is not the
norm on our fs. GPFS access continues to be ok during these NFS/SMB hangs.
Another problem it causes is that snapshots on the root fileset cannot be
created while this global snapshot is being deleted. This disrupts our
daily snapshot creation schedule and I also have to switch to running
mmbackup without a snapshot, since I won't be able to create a new global
snapshot if another global snapshot is being deleted.

If possible I'd request that running multiple snapshot related commands
simultaneously be allowed, at least the ability to run an mmcrsnapshot
while an mmdelsnapshot is running would be nice. I can see that running
multiple global snapshot deletions or multiple simultaneous snapshot
deletions on the same fileset would be risky/very difficult (or impossible)
to implement but it seems like it should be possible to "quiesce" an
mmdelsnapshot for a few seconds so that an mmcrsnapshot on the same fileset
can be completed. This can be done manually by interrupting an ongoing
snapshot deletion, creating the new snapshot, and continuing the deletion
but would be nice if this could be fully automated . What would be even
cooler would be a snapshot command queuing system for deletion requests
within the same fileset, that way snapshot deletions would be queued up and
automatically occur as soon as deletions in the same fileset are complete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140527/edbbdf15/attachment-0001.htm>

From sabujp at gmail.com  Fri May 30 02:34:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 29 May 2014 20:34:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
Message-ID: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>

This is still happening in 3.5.0.18 and when a snapshot is being deleted it
slows NFS read speeds to a crawl (but not gpfs and not NFS writes).


On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

> Hi all,
>
> We're running 3.5.0.17 now and it looks like the filesystem manager
> automatically reboots (and sometimes fails to automatically reboot) after
> mmdelsnapshot is called, either from the filesystem manager itself or from
> some other nsd/node . It didn't start happening immediately after we
> updated to 17, but we never had this issue when we were at 3.5.0.11 . The
> error mmdelsnapshot throws at some point is :
>
> Lost connection to file system daemon.
> mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
> mmdelsnapshot: Command failed.  Examine previous error messages to
> determine cause.
>
> It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.
>
>
> It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :
>
>
> May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
> Address already in use
>
>
> May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port
>
> for RPC socket: Name or service not known
>
>
> Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?
>
>
> Thanks,
>
> Sabuj
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140529/76d1258b/attachment-0001.htm>

From bbanister at jumptrading.com  Fri May 30 14:35:55 2014
From: bbanister at jumptrading.com (Bryan Banister)
Date: Fri, 30 May 2014 13:35:55 +0000
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>

This sounds like a serious problem and you should open a PMR with IBM to get direct guidance.

I normally will take a GPFS trace during a problem like this from all of the nodes that are affected or directly involved in the operation.

Hope that helps,
-Bryan

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Sabuj Pattanayek
Sent: Thursday, May 29, 2014 8:34 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] filesystem manager crashes every time mmdelsnapshot (from either the filesystem manager or some other nsd/client) is called

This is still happening in 3.5.0.18 and when a snapshot is being deleted it slows NFS read speeds to a crawl (but not gpfs and not NFS writes).

On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com<mailto:sabujp at gmail.com>> wrote:
Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager automatically reboots (and sometimes fails to automatically reboot) after mmdelsnapshot is called, either from the filesystem manager itself or from some other nsd/node . It didn't start happening immediately after we updated to 17, but we never had this issue when we were at 3.5.0.11 . The error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.

mmdelsnapshot: An internode connection between GPFS nodes was disrupted.

mmdelsnapshot: Command failed.  Examine previous error messages to

determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.



It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :



May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:

Address already in use



May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known



Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?



Thanks,

Sabuj



________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/d06dcb69/attachment-0001.htm>

From sabujp at gmail.com  Fri May 30 14:49:15 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 30 May 2014 08:49:15 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
	<21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <CAEeMGHuGfMrVPYd97wPQ_FfCj=ocEzFb7fP9Rx0zQC623==b=A@mail.gmail.com>

On Fri, May 30, 2014 at 8:35 AM, Bryan Banister <bbanister at jumptrading.com>
wrote:

>  This sounds like a serious problem and you should open a PMR with IBM to
> get direct guidance.
>
>
>
> I normally will take a GPFS trace during a problem like this from all of
> the nodes that are affected or directly involved in the operation.
>

easier said than done, I've tried at least 10 times. It's almost as if the
act of running the mmtrace prevents the error from occurring, we've sent
internaldumps and gpfs.snap that were generated to ddn. Not having much
luck in figuring out if it's been sent through to IBM yet or not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/dd6729bd/attachment-0001.htm>

From chair at gpfsug.org  Fri May  2 15:49:54 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Fri, 02 May 2014 15:49:54 +0100
Subject: [gpfsug-discuss] GPFS UG 10 Presentations available
Message-ID: <5363B092.8020507@gpfsug.org>

Hello all

   Firstly, thanks for the feedback we've had so far.  Very much 
appreciated.

Secondly, GPFS UG 10 Presentations are now available on the 
Presentations section of the website.
Any outstanding presentations will follow shortly.

See: http://www.gpfsug.org/

Best regards,

Jez

UG Chair


From oester at gmail.com  Fri May  2 16:06:39 2014
From: oester at gmail.com (Bob Oesterlin)
Date: Fri, 2 May 2014 10:06:39 -0500
Subject: [gpfsug-discuss] GPFS UG 10 Presentations - Sven Oehme
Message-ID: <CAMNdFvAjd=oNzygAT2MDF2B1Ps6+VcALjOWu-TUywm2Q1ytd7w@mail.gmail.com>

It Sven's presentation, he mentions a tools "trcio" (in
/xcat/oehmes/gpfs-clone)

Where can I find that?

Bob Oesterlin



On Fri, May 2, 2014 at 9:49 AM, Jez Tucker (Chair) <chair at gpfsug.org> wrote:

> Hello all
>
>   Firstly, thanks for the feedback we've had so far.  Very much
> appreciated.
>
> Secondly, GPFS UG 10 Presentations are now available on the Presentations
> section of the website.
> Any outstanding presentations will follow shortly.
>
> See: http://www.gpfsug.org/
>
> Best regards,
>
> Jez
>
> UG Chair
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140502/dc657ca8/attachment-0002.htm>

From aortiz at serviware.com  Mon May  5 10:49:53 2014
From: aortiz at serviware.com (Aurelien Ortiz  ( Serviware Toulouse ))
Date: Mon, 5 May 2014 11:49:53 +0200
Subject: [gpfsug-discuss] Hello !
Message-ID: <2F5DD237-3157-4941-B110-3F92E8ED1B4D@serviware.fr>

Hello, 

my name is Aur?lien Ortiz. 
I work at Serviware which is a subsidiary of Bull (french manufacturer). 

Serviware sells and installs clusters of small and medium size, with GPFS as the main storage solution. We use it either as a fast scratch space or a more reliable storage space in order to store results of computation.

Some of our customers : PSA, TOTAL, Renault, EDF, CNRS, CEA, IPGP, IPA, Cenaero, IFPEN, INRIA, etc.

I'm looking forward to share about GPFS with you.

Regards,
?
Aur?lien Ortiz
HPC engineer
Serviware / BULL



From luke.raimbach at oerc.ox.ac.uk  Wed May  7 10:28:59 2014
From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach)
Date: Wed, 7 May 2014 09:28:59 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <F68FAAD16AEC9744BD921F91014D2F00371189@MBX06.ad.oak.ox.ac.uk>

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639




From Paul.Sanchez at deshaw.com  Wed May  7 11:59:30 2014
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 7 May 2014 10:59:30 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <201D6001C896B846A9CFC2E841986AC11A259268@mailnycmb2a.winmail.deshaw.com>

Hi Luke,

When using RFC 1918 space among remote clusters, GPFS assumes that each cluster's privately addressed networks are not reachable from one another.  You must add explicit shared subnets via mmchconfig. Try setting subnets as follows:

gpfs.oerc.local:
subnets="10.200.0.0 10.200.0.0/cpdn.oerc.local"

cpdn.oerc.local:
subnets="10.200.0.0 10.200.0.0/gpfs.oerc.local"

I think you may also need to set the cipher list locally on each cluster to AUTHONLY via mmauth. On my clusters, these match. (No cluster says "none specified".)

Hope that helps,
Paul



Sent with Good (www.good.com)

________________________________
From: gpfsug-discuss-bounces at gpfsug.org on behalf of Luke Raimbach
Sent: Wednesday, May 07, 2014 5:28:59 AM
To: gpfsug-discuss at gpfsug.org
Subject: [gpfsug-discuss] GPFS Remote Mount Fails

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140507/f289cd64/attachment-0002.htm>

From pavel.pokorny at datera.cz  Fri May  9 10:15:54 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Fri, 9 May 2014 17:15:54 +0800
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
Message-ID: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>

Hello,
our customer would like to deny some of the file extensions (.mp3, .avi,
...) to be stored on GPFS FS. Is there any way to deny some of the
extensions?
Right now we only find the solution to make a placement rule that directs
the files to metadataonly storage pool.

This leads me to second question. Is there any easy way to make a group of
extension (some type of templates) for placement rules? Because there is a
lot of extensions, customer would like to deny, and the placement rules
files start to grow :-(.

Thank you very much for your ideas and answers.
Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140509/b151676f/attachment-0002.htm>

From jonathan at buzzard.me.uk  Fri May  9 14:21:58 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Fri, 09 May 2014 14:21:58 +0100
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
In-Reply-To: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
References: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
Message-ID: <1399641718.19065.22.camel@buzzard.phy.strath.ac.uk>

On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> Hello,
> our customer would like to deny some of the file extensions
> (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> some of the extensions?
> Right now we only find the solution to make a placement rule that
> directs the files to metadataonly storage pool.
> 

Create a really small storage pool and fill it in advance.

> 
> This leads me to second question. Is there any easy way to make a
> group of extension (some type of templates) for placement rules?
> Because there is a lot of extensions, customer would like to deny, and
> the placement rules files start to grow :-(.

Note if you deny extensions out right you are more likely to force users
to come up with work arounds. The simple expediency of changing the file
extension for example will lead you to a game of whack-a-mole.

Better to given them really rubbish performance and don't talk about it.
The chances are this is less of a business cost than dealing with the
outright banning, especially when there turns out to be some legitimate
business reason for having multimedia stuff, if even only occasionally.

So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
files. Performance will suck, but the users probably won't realize what
is going on, and are unlikely to come to support asking why their itunes
collection is really slow. That said if you are doing ILM with a
nearline pool holding most of the storage, it is likely to be near idle
most of the time so just force all multimedia stuff to your nearline
pool is adequate.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From pavel.pokorny at datera.cz  Mon May 12 03:46:40 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Mon, 12 May 2014 10:46:40 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
Message-ID: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>

Hello,
after discussion with Marc A Kaplan from IBM I finished with this solution.
So I am sharing it with all other user just in case you need it in future.

Example placement policy file:

/* --- Enable .avi so it will not be denied by default --- */
RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$'])

/* --- Deny all following extensions --- */
RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|\.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|\.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|\.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|\.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|\.if$|\.spiff$|\.tif*$'])

/* --- Enable every other file extensions --- */
RULE 'DEFAULT' SET POOL 'data'

Bye, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Sat, May 10, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: GPFS Placement rules - group file extension?
>       (Jonathan Buzzard)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 09 May 2014 14:21:58 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] GPFS Placement rules - group file
>         extension?
> Message-ID: <1399641718.19065.22.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> > Hello,
> > our customer would like to deny some of the file extensions
> > (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> > some of the extensions?
> > Right now we only find the solution to make a placement rule that
> > directs the files to metadataonly storage pool.
> >
>
> Create a really small storage pool and fill it in advance.
>
> >
> > This leads me to second question. Is there any easy way to make a
> > group of extension (some type of templates) for placement rules?
> > Because there is a lot of extensions, customer would like to deny, and
> > the placement rules files start to grow :-(.
>
> Note if you deny extensions out right you are more likely to force users
> to come up with work arounds. The simple expediency of changing the file
> extension for example will lead you to a game of whack-a-mole.
>
> Better to given them really rubbish performance and don't talk about it.
> The chances are this is less of a business cost than dealing with the
> outright banning, especially when there turns out to be some legitimate
> business reason for having multimedia stuff, if even only occasionally.
>
> So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
> files. Performance will suck, but the users probably won't realize what
> is going on, and are unlikely to come to support asking why their itunes
> collection is really slow. That said if you are doing ILM with a
> nearline pool holding most of the storage, it is likely to be near idle
> most of the time so just force all multimedia stuff to your nearline
> pool is adequate.
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 6
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140512/84920933/attachment-0002.htm>

From jonathan at buzzard.me.uk  Mon May 12 12:12:19 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Mon, 12 May 2014 12:12:19 +0100
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
Message-ID: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>

On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> Hello,
> after discussion with Marc A Kaplan from IBM I finished with this
> solution. So I am sharing it with all other user just in case you need
> it in future.
> 
> 
> Example placement policy file:
> 
> 
> /* --- Enable .avi so it will not be denied by default --- */
> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$'])
> 
> 
> /* --- Deny all following extensions --- */
> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> \.if$|\.spiff$|\.tif*$'])
> 

To demonstrate why this is a wack-a-mole exercise, as a user of your
system I will just save in iPod friendly format, all my audio AAC
encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
audiobooks in .m4b format :-)

I would further note that it is rather trivial to pick a random
extension say .sdf and associate that with just about any program, most
of which examine the contents of the file and don't rely on the
extension.

Further noting if you are continuing in your wack-a-mole exercise I
could use .divx which you have conveniently not blocked.

Finally being the resourceful sort I will just embed it all in a PDF, or
word document, or powerpoint, or ...

I don't know exactly what your customer is expecting, but this sounds
like a management idea, with little understanding of how things work in
the real world.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From zgiles at gmail.com  Mon May 12 15:02:04 2014
From: zgiles at gmail.com (Zachary Giles)
Date: Mon, 12 May 2014 10:02:04 -0400
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
	<1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAMYZk=f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg@mail.gmail.com>

To further the wack-a-mole problem:
If you set those extensions to be in the system pool and assume the
system pool has no data-disks, it is still possible for GPFS 3.5 to
create a file in the system pool if one is utilizing the
small-files-in-metadata feature (whatever it's called now) and if the
file is 0-3.9K in size where it will be stored in the xattrs.
So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
if, say, you wanted to restrict .txt files, which are often that
small.



On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
<jonathan at buzzard.me.uk> wrote:
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
>> Hello,
>> after discussion with Marc A Kaplan from IBM I finished with this
>> solution. So I am sharing it with all other user just in case you need
>> it in future.
>>
>>
>> Example placement policy file:
>>
>>
>> /* --- Enable .avi so it will not be denied by default --- */
>> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$'])
>>
>>
>> /* --- Deny all following extensions --- */
>> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
>> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
>> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
>> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
>> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
>> \.if$|\.spiff$|\.tif*$'])
>>
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-- 
Zach Giles
zgiles at gmail.com


From pavel.pokorny at datera.cz  Tue May 13 15:35:25 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Tue, 13 May 2014 22:35:25 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28,
	Issue 8 (Pavel Pokorny)
Message-ID: <CAHy3nHL4t0vAC8c7PuFwsa6Wo-T3VCTNSpFc0D=-G9Su4MwrXw@mail.gmail.com>

Hello,
just to clarify. Customer need is very easy: Deny storing selected files
extension in specific filesets.
I also don't like the way to direct placement to metadata only pool. I also
know that small filse can actually be stored there.
I will appreciate more help how to make it. I don't think that expalanation
to customer that his need is stupid, is a good approach to the customer.
If there is or not .divx extension is not important, we care about GPFS
stuff and the extensions were selected by customer.
Thank you, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Tue, May 13, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Jonathan Buzzard)
>    2. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Zachary Giles)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 12 May 2014 12:12:19 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID: <1399893139.27622.11.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> > Hello,
> > after discussion with Marc A Kaplan from IBM I finished with this
> > solution. So I am sharing it with all other user just in case you need
> > it in future.
> >
> >
> > Example placement policy file:
> >
> >
> > /* --- Enable .avi so it will not be denied by default --- */
> > RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$'])
> >
> >
> > /* --- Deny all following extensions --- */
> > RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> > \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> > \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> > \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> > \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> > \.if$|\.spiff$|\.tif*$'])
> >
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 12 May 2014 10:02:04 -0400
> From: Zachary Giles <zgiles at gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID:
>         <CAMYZk=
> f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> To further the wack-a-mole problem:
> If you set those extensions to be in the system pool and assume the
> system pool has no data-disks, it is still possible for GPFS 3.5 to
> create a file in the system pool if one is utilizing the
> small-files-in-metadata feature (whatever it's called now) and if the
> file is 0-3.9K in size where it will be stored in the xattrs.
> So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
> if, say, you wanted to restrict .txt files, which are often that
> small.
>
>
>
> On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
> <jonathan at buzzard.me.uk> wrote:
> > On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> >> Hello,
> >> after discussion with Marc A Kaplan from IBM I finished with this
> >> solution. So I am sharing it with all other user just in case you need
> >> it in future.
> >>
> >>
> >> Example placement policy file:
> >>
> >>
> >> /* --- Enable .avi so it will not be denied by default --- */
> >> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$'])
> >>
> >>
> >> /* --- Deny all following extensions --- */
> >> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> >> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> >> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> >> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> >> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> >> \.if$|\.spiff$|\.tif*$'])
> >>
> >
> > To demonstrate why this is a wack-a-mole exercise, as a user of your
> > system I will just save in iPod friendly format, all my audio AAC
> > encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> > audiobooks in .m4b format :-)
> >
> > I would further note that it is rather trivial to pick a random
> > extension say .sdf and associate that with just about any program, most
> > of which examine the contents of the file and don't rely on the
> > extension.
> >
> > Further noting if you are continuing in your wack-a-mole exercise I
> > could use .divx which you have conveniently not blocked.
> >
> > Finally being the resourceful sort I will just embed it all in a PDF, or
> > word document, or powerpoint, or ...
> >
> > I don't know exactly what your customer is expecting, but this sounds
> > like a management idea, with little understanding of how things work in
> > the real world.
> >
> >
> > JAB.
> >
> > --
> > Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> > Fife, United Kingdom.
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> --
> Zach Giles
> zgiles at gmail.com
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 8
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140513/a9784129/attachment-0002.htm>

From sabujp at gmail.com  Thu May 15 13:48:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 15 May 2014 07:48:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
Message-ID: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>

Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager
automatically reboots (and sometimes fails to automatically reboot) after
mmdelsnapshot is called, either from the filesystem manager itself or from
some other nsd/node . It didn't start happening immediately after we
updated to 17, but we never had this issue when we were at 3.5.0.11 . The
error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.
mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
mmdelsnapshot: Command failed.  Examine previous error messages to
determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup
- CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to
not reboot itself (which is actually worse), but the former does.


It also causes some havoc with CNFS file locking even after the
filesystem manager is rebooted and has come up :


May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
Address already in use


May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known


Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior
or know if it's fixed in 18?


Thanks,

Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140515/f6db747b/attachment-0002.htm>

From sabujp at gmail.com  Mon May 19 16:16:14 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Mon, 19 May 2014 10:16:14 -0500
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/09e7cd91/attachment-0002.htm>

From TOMP at il.ibm.com  Mon May 19 16:35:32 2014
From: TOMP at il.ibm.com (Tomer Perry)
Date: Mon, 19 May 2014 18:35:32 +0300
Subject: [gpfsug-discuss] gpfs 4.1 features?
In-Reply-To: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
References: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
Message-ID: <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702@il.ibm.com>

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm

Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/8d477962/attachment-0002.htm>

From johnlang at us.ibm.com  Tue May 20 13:00:19 2014
From: johnlang at us.ibm.com (John Langlois)
Date: Tue, 20 May 2014 08:00:19 -0400
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
Message-ID: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>

Folks,

We're going through a naming transition and you will see some GPFS 
features now described under the codename Elastic Storage.  Don't let the 
codename confuse you ... I'll let you all know the moment we get the final 
Family and Product naming straight. We are changing the name to reflect 
the fact that IBM is significantly increasing investment in this 
technology area and we will stretch GPFS out into the Cloud with Open 
Stack object support (e.g. SWIFT object store guide coming in June).  In 
addition, GPFS will be available in the Softlayer Cloud Managed Service 
Offering catalog soon.  You will be seeing a lot of new and exciting news 
on our roadmaps.

New to GPFS 4.1 features include:
Native encryption and secure cryptography erase (data wiping) that is NIST 
Compliant and FIPS certified.
GPFS client-side IBM Elastic Storage Flash caches that increase I/O 
performance up to 6x
Active File Manager now supports parallel data transfers between sites, 
using multiple gateway nodes 
AFM now supports the GPFS protocol in addition to NFS protocol for file 
transfer
Much simpler socket based pricing - no more PVUs. 
Lots of Usability improvements that include
  a.  NFS Data Migration - Eases data transfer from NFS when upgrading 
hardware or buying a new system 
  b   AFM Optimization  - Improved prefetch performance; support for the 
native GPFS protocol   (described above)
  c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool; 
Backup utility enhancements
  d.  FPO Improvements - Data locality aware file system recovery and 
improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a 
Cinder driver that gives architects access to GPFS features and 
capabilities.

Here's the current external website landing page:  
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager

Phone: 1-919-267-6767
E-mail: johnlang at us.ibm.com
Find me on:   and within IBM on:  






From:   gpfsug-discuss-request at gpfsug.org
To:     gpfsug-discuss at gpfsug.org
Date:   05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                 gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                 http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                 gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                 gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. gpfs 4.1 features? (Sabuj Pattanayek)
   2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0006.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0007.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0008.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0002.gif>

From sjhoward at iu.edu  Tue May 20 15:17:32 2014
From: sjhoward at iu.edu (Howard, Stewart Jameson)
Date: Tue, 20 May 2014 14:17:32 +0000
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
Message-ID: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>

Hi All,

My name is Stewart Howard and I work for Indiana University as an admin on a two-site replicated GPFS cluster.  I'm a new member of this mailing list and this is my first post  :)

Recently, we've discovered that small-file performance on our system is pretty lack-luster.  For comparison, here are some numbers:

1)  When transferring large files (~2 GB), we get outstanding performance and can typically saturate the client's network connection.  We generally see about 490 MB/s over a 10Gb line, which should be about right, given that we lose half of our bandwidth to replication.

2)  When transferring a large number of small files, we get a very poor transfer rate, generally on the order of 2 MB/s, writing from a client node *inside* the GPFS cluster.

I'm wondering if anyone else has experience with similar performance issues and what ended up being the cause/solution.  Also, I would be interested in hearing any general rules-of-thumb that the group has found helpful in balancing performance between large-file and small-file I/O.

We have gathered some diagnostic information while performing various small-file I/O operations, as well as a variety of metadata operations in quick succession.  I'd be happy to share results of the diagnostics, if it would help provide context.

Thank you so much for all of your help!

Stewart Howard
Indiana University


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/63b40754/attachment-0002.htm>

From dhildeb at us.ibm.com  Tue May 20 18:29:51 2014
From: dhildeb at us.ibm.com (Dean Hildebrand)
Date: Tue, 20 May 2014 10:29:51 -0700
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
	<OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
Message-ID: <OF2F748B64.B172A915-ON88257CDE.00601714-88257CDE.00601DF1@us.ibm.com>


Nice email!

Dean Hildebrand
IBM Master Inventor and Manager | Scale-out Storage Software
IBM Almaden Research Center




From:	John Langlois/Raleigh/IBM at IBMUS
To:	gpfsug-discuss at gpfsug.org
Date:	05/20/2014 05:01 AM
Subject:	[gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
Sent by:	gpfsug-discuss-bounces at gpfsug.org



Folks,

We're going through a naming transition and you will see some GPFS features
now described under the codename Elastic Storage.  Don't let the codename
confuse you ... I'll let you all know the moment we get the final Family
and Product naming straight. We are changing the name to reflect the fact
that IBM is significantly increasing investment in this technology area and
we will stretch GPFS out into the Cloud with Open Stack object support
(e.g. SWIFT object store guide coming in June).  In addition, GPFS will be
available in the Softlayer Cloud Managed Service Offering catalog soon.
You will be seeing a lot of new and exciting news on our roadmaps.

New to GPFS 4.1 features include:
   1.	Native encryption and secure cryptography erase (data wiping) that is
      NIST Compliant and FIPS certified.
   2.	GPFS client-side IBM Elastic Storage Flash caches that increase I/O
      performance up to 6x
   3.	Active File Manager now supports parallel data transfers between
      sites, using multiple gateway nodes
   4.	AFM now supports the GPFS protocol in addition to NFS protocol for
      file transfer
   5.	Much simpler socket based pricing - no more PVUs.
   6.	Lots of Usability improvements that include
       a.  NFS Data Migration - Eases data transfer from NFS when upgrading
      hardware or buying a new system
       b   AFM Optimization  - Improved prefetch performance; support for
      the native GPFS protocol   (described above)
       c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool;
      Backup utility enhancements
       d.  FPO Improvements - Data locality aware file system recovery and
      improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a
Cinder driver that gives architects access to GPFS features and
capabilities.

Here's the current external website landing page:
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager
                                                                                                                                      
                                                                                                                                      
                                                                                                                                      
 Phone: 1-919-267-6767                                                                                                            IBM 
 E-mail: johnlang at us.ibm.com                                                                                                          
 Find me on: LinkedIn:                                                                                                                
 http://www.linkedin.com/profile/view?id=24557762&authType=NAME_SEARCH&authToken=b9zY&locale=en_US&sr                                 
 Twitter: https://twitter.com/johnlang123 and within IBM on: IBM Connections:                                                         
 https://w3-connections.ibm.com/profiles/html/profileView.do?userid=69bf0ec0-8f0a-1028-94ba-db07163b5                                 
                                                                                                                                      
                                                                                                                                      






From:        gpfsug-discuss-request at gpfsug.org
To:        gpfsug-discuss at gpfsug.org
Date:        05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

  1. gpfs 4.1 features? (Sabuj Pattanayek)
  2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0004.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62923814.jpg
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0006.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62235135.jpg
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0007.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62861266.jpg
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0008.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62781961.gif
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0005.gif>

From chekh at stanford.edu  Wed May 21 21:04:52 2014
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 21 May 2014 13:04:52 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
Message-ID: <537D06E4.3090603@stanford.edu>

Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860


From oehmes at us.ibm.com  Wed May 21 22:42:51 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Wed, 21 May 2014 14:42:51 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <537D06E4.3090603@stanford.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
	<537D06E4.3090603@stanford.edu>
Message-ID: <OF11EC8E07.09F9EDEE-ON88257CDF.0074EF02-88257CDF.007747E9@us.ibm.com>

Hi Alex, Stewart,

the problem is more likely a latency and lack of parallelism of the 
transfer.
 
if you have 1 thread that transfers files of 1k in size you don't get a 
lot of bandwidth as it transfers one at a time and the latency kills the 
bandwidth. to explain this on an example :

assume your network between client and server is 10 gigabit and both nodes 
are capable of pushing this.

if it takes 1 ms for reading + writing each 1 kb file you get ~ 1.02 
MB/sec
if your filesize changes to 4k, even you don't do anything else it goes up 
to 4.06 MB/sec
if you can reduce latency on read or write to lets say 100us the same 
process would transfer 37.86 MB/sec

so this is a latency / parallelism problem and this has nothing to do with 
GPFS, you would have exactly the same issue.
the solution is to copy the data with a tool that does it multi threaded 
as the numbers above are based on 1 thread. 
if you would have 100us read+write time and transfer 4k files with 10 
threads in parallel the same transfer would be close to 400 MB/sec.

hope that helps. Sven 
------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
Phone: +1 (408) 824-8904 
IBM Almaden Research Lab 
------------------------------------------



From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org
Date:   05/21/2014 01:05 PM
Subject:        Re: [gpfsug-discuss] Mitigating Poor Small-file I/O 
Performance
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140521/08d101d4/attachment-0002.htm>

From sabujp at gmail.com  Tue May 27 22:29:02 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Tue, 27 May 2014 16:29:02 -0500
Subject: [gpfsug-discuss] feature request : ability to run multiple snapshot
	related commands simultaneously
Message-ID: <CAEeMGHuFnKtH1a8eirKqu5r5wxeOfC_wVahpwSnPio8k8XV51w@mail.gmail.com>

Sometimes snapshot deletions can take a very long time. While they're
running no other snapshot related commands can be run, e.g. it doesn't look
like I can run an mmcrsnapshot on the same fileset while a global snapshot
deletion or a snapshot deletion on the same fileset is ongoing.

Today's snapshot deletion on a global snapshot created about 19 hours ago
is going painfully slowly for some reason. It's also affecting CNFS and SMB
performance with periods of hangs for about 30+ seconds. This is not the
norm on our fs. GPFS access continues to be ok during these NFS/SMB hangs.
Another problem it causes is that snapshots on the root fileset cannot be
created while this global snapshot is being deleted. This disrupts our
daily snapshot creation schedule and I also have to switch to running
mmbackup without a snapshot, since I won't be able to create a new global
snapshot if another global snapshot is being deleted.

If possible I'd request that running multiple snapshot related commands
simultaneously be allowed, at least the ability to run an mmcrsnapshot
while an mmdelsnapshot is running would be nice. I can see that running
multiple global snapshot deletions or multiple simultaneous snapshot
deletions on the same fileset would be risky/very difficult (or impossible)
to implement but it seems like it should be possible to "quiesce" an
mmdelsnapshot for a few seconds so that an mmcrsnapshot on the same fileset
can be completed. This can be done manually by interrupting an ongoing
snapshot deletion, creating the new snapshot, and continuing the deletion
but would be nice if this could be fully automated . What would be even
cooler would be a snapshot command queuing system for deletion requests
within the same fileset, that way snapshot deletions would be queued up and
automatically occur as soon as deletions in the same fileset are complete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140527/edbbdf15/attachment-0002.htm>

From sabujp at gmail.com  Fri May 30 02:34:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 29 May 2014 20:34:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
Message-ID: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>

This is still happening in 3.5.0.18 and when a snapshot is being deleted it
slows NFS read speeds to a crawl (but not gpfs and not NFS writes).


On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

> Hi all,
>
> We're running 3.5.0.17 now and it looks like the filesystem manager
> automatically reboots (and sometimes fails to automatically reboot) after
> mmdelsnapshot is called, either from the filesystem manager itself or from
> some other nsd/node . It didn't start happening immediately after we
> updated to 17, but we never had this issue when we were at 3.5.0.11 . The
> error mmdelsnapshot throws at some point is :
>
> Lost connection to file system daemon.
> mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
> mmdelsnapshot: Command failed.  Examine previous error messages to
> determine cause.
>
> It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.
>
>
> It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :
>
>
> May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
> Address already in use
>
>
> May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port
>
> for RPC socket: Name or service not known
>
>
> Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?
>
>
> Thanks,
>
> Sabuj
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140529/76d1258b/attachment-0002.htm>

From bbanister at jumptrading.com  Fri May 30 14:35:55 2014
From: bbanister at jumptrading.com (Bryan Banister)
Date: Fri, 30 May 2014 13:35:55 +0000
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>

This sounds like a serious problem and you should open a PMR with IBM to get direct guidance.

I normally will take a GPFS trace during a problem like this from all of the nodes that are affected or directly involved in the operation.

Hope that helps,
-Bryan

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Sabuj Pattanayek
Sent: Thursday, May 29, 2014 8:34 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] filesystem manager crashes every time mmdelsnapshot (from either the filesystem manager or some other nsd/client) is called

This is still happening in 3.5.0.18 and when a snapshot is being deleted it slows NFS read speeds to a crawl (but not gpfs and not NFS writes).

On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com<mailto:sabujp at gmail.com>> wrote:
Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager automatically reboots (and sometimes fails to automatically reboot) after mmdelsnapshot is called, either from the filesystem manager itself or from some other nsd/node . It didn't start happening immediately after we updated to 17, but we never had this issue when we were at 3.5.0.11 . The error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.

mmdelsnapshot: An internode connection between GPFS nodes was disrupted.

mmdelsnapshot: Command failed.  Examine previous error messages to

determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.



It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :



May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:

Address already in use



May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known



Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?



Thanks,

Sabuj



________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/d06dcb69/attachment-0002.htm>

From sabujp at gmail.com  Fri May 30 14:49:15 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 30 May 2014 08:49:15 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
	<21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <CAEeMGHuGfMrVPYd97wPQ_FfCj=ocEzFb7fP9Rx0zQC623==b=A@mail.gmail.com>

On Fri, May 30, 2014 at 8:35 AM, Bryan Banister <bbanister at jumptrading.com>
wrote:

>  This sounds like a serious problem and you should open a PMR with IBM to
> get direct guidance.
>
>
>
> I normally will take a GPFS trace during a problem like this from all of
> the nodes that are affected or directly involved in the operation.
>

easier said than done, I've tried at least 10 times. It's almost as if the
act of running the mmtrace prevents the error from occurring, we've sent
internaldumps and gpfs.snap that were generated to ddn. Not having much
luck in figuring out if it's been sent through to IBM yet or not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/dd6729bd/attachment-0002.htm>

From chair at gpfsug.org  Fri May  2 15:49:54 2014
From: chair at gpfsug.org (Jez Tucker (Chair))
Date: Fri, 02 May 2014 15:49:54 +0100
Subject: [gpfsug-discuss] GPFS UG 10 Presentations available
Message-ID: <5363B092.8020507@gpfsug.org>

Hello all

   Firstly, thanks for the feedback we've had so far.  Very much 
appreciated.

Secondly, GPFS UG 10 Presentations are now available on the 
Presentations section of the website.
Any outstanding presentations will follow shortly.

See: http://www.gpfsug.org/

Best regards,

Jez

UG Chair


From oester at gmail.com  Fri May  2 16:06:39 2014
From: oester at gmail.com (Bob Oesterlin)
Date: Fri, 2 May 2014 10:06:39 -0500
Subject: [gpfsug-discuss] GPFS UG 10 Presentations - Sven Oehme
Message-ID: <CAMNdFvAjd=oNzygAT2MDF2B1Ps6+VcALjOWu-TUywm2Q1ytd7w@mail.gmail.com>

It Sven's presentation, he mentions a tools "trcio" (in
/xcat/oehmes/gpfs-clone)

Where can I find that?

Bob Oesterlin



On Fri, May 2, 2014 at 9:49 AM, Jez Tucker (Chair) <chair at gpfsug.org> wrote:

> Hello all
>
>   Firstly, thanks for the feedback we've had so far.  Very much
> appreciated.
>
> Secondly, GPFS UG 10 Presentations are now available on the Presentations
> section of the website.
> Any outstanding presentations will follow shortly.
>
> See: http://www.gpfsug.org/
>
> Best regards,
>
> Jez
>
> UG Chair
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140502/dc657ca8/attachment-0003.htm>

From aortiz at serviware.com  Mon May  5 10:49:53 2014
From: aortiz at serviware.com (Aurelien Ortiz  ( Serviware Toulouse ))
Date: Mon, 5 May 2014 11:49:53 +0200
Subject: [gpfsug-discuss] Hello !
Message-ID: <2F5DD237-3157-4941-B110-3F92E8ED1B4D@serviware.fr>

Hello, 

my name is Aur?lien Ortiz. 
I work at Serviware which is a subsidiary of Bull (french manufacturer). 

Serviware sells and installs clusters of small and medium size, with GPFS as the main storage solution. We use it either as a fast scratch space or a more reliable storage space in order to store results of computation.

Some of our customers : PSA, TOTAL, Renault, EDF, CNRS, CEA, IPGP, IPA, Cenaero, IFPEN, INRIA, etc.

I'm looking forward to share about GPFS with you.

Regards,
?
Aur?lien Ortiz
HPC engineer
Serviware / BULL



From luke.raimbach at oerc.ox.ac.uk  Wed May  7 10:28:59 2014
From: luke.raimbach at oerc.ox.ac.uk (Luke Raimbach)
Date: Wed, 7 May 2014 09:28:59 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <F68FAAD16AEC9744BD921F91014D2F00371189@MBX06.ad.oak.ox.ac.uk>

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639




From Paul.Sanchez at deshaw.com  Wed May  7 11:59:30 2014
From: Paul.Sanchez at deshaw.com (Sanchez, Paul)
Date: Wed, 7 May 2014 10:59:30 +0000
Subject: [gpfsug-discuss] GPFS Remote Mount Fails
Message-ID: <201D6001C896B846A9CFC2E841986AC11A259268@mailnycmb2a.winmail.deshaw.com>

Hi Luke,

When using RFC 1918 space among remote clusters, GPFS assumes that each cluster's privately addressed networks are not reachable from one another.  You must add explicit shared subnets via mmchconfig. Try setting subnets as follows:

gpfs.oerc.local:
subnets="10.200.0.0 10.200.0.0/cpdn.oerc.local"

cpdn.oerc.local:
subnets="10.200.0.0 10.200.0.0/gpfs.oerc.local"

I think you may also need to set the cipher list locally on each cluster to AUTHONLY via mmauth. On my clusters, these match. (No cluster says "none specified".)

Hope that helps,
Paul



Sent with Good (www.good.com)

________________________________
From: gpfsug-discuss-bounces at gpfsug.org on behalf of Luke Raimbach
Sent: Wednesday, May 07, 2014 5:28:59 AM
To: gpfsug-discuss at gpfsug.org
Subject: [gpfsug-discuss] GPFS Remote Mount Fails

Dear All,

I'm having a problem remote mounting a file system. I have two clusters:

gpfs.oerc.local which owns file system 'gpfs'
cpdn.oerc.local which owns no file systems

I want to remote mount file system 'gpfs' from cluster cpdn.oerc.local. I'll post the configuration for both clusters further down. The error I receive on a node in cluster cpdn.oerc.local is:


Wed May  7 10:05:19.595 2014: Waiting to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.598 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.599 2014: Remote mounts are not enabled within this cluster.
Wed May  7 10:05:20.598 2014: A node join was rejected.  This could be due to
incompatible daemon versions, failure to find the node
in the configuration database, or no configuration manager found.
Wed May  7 10:05:20.600 2014: Failed to join remote cluster gpfs.oerc.local
Wed May  7 10:05:20.601 2014: Command: err 693: mount gpfs.oerc.local:gpfs
Wed May  7 10:05:20.600 2014: Message failed because the destination node refused the connection.


I'm concerned about the "Remote mounts are not enabled within this cluster" messages. Having followed the configuration steps in the GPFS Advanced Administration Guide, I end up with the following configurations:

## GPFS Cluster 'gpfs.oerc.local' ##

[root at gpfs01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs.oerc.local
  GPFS cluster id:           748734524680043237
  GPFS UID domain:           gpfs.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    gpfs01.oerc.local
  Secondary server:  gpfs02.oerc.local

 Node  Daemon node name    IP address     Admin node name     Designation
--------------------------------------------------------------------------
   1   gpfs01.oerc.local   10.100.10.21   gpfs01.oerc.local   quorum-manager
   2   gpfs02.oerc.local   10.100.10.22   gpfs02.oerc.local   quorum-manager
   3   linux.oerc.local    10.100.10.1    linux.oerc.local
   4   jupiter.oerc.local  10.100.10.2    jupiter.oerc.local
   5   cnfs0.oerc.local    10.100.10.100  cnfs0.oerc.local
   6   cnfs1.oerc.local    10.100.10.101  cnfs1.oerc.local
   7   cnfs2.oerc.local    10.100.10.102  cnfs2.oerc.local
   8   cnfs3.oerc.local    10.100.10.103  cnfs3.oerc.local
   9   tsm01.oerc.local    10.100.10.51   tsm01.oerc.local    quorum-manager


[root at gpfs01 ~]# mmremotecluster show all
Cluster name:    cpdn.oerc.local
Contact nodes:   10.100.10.60,10.100.10.61,10.100.10.62
SHA digest:      e9a2dc678a62d6c581de0b89b49a90f28f401327
File systems:    (none defined)


[root at gpfs01 ~]# mmauth show all
Cluster name:        cpdn.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  gpfs      (rw, root remapped to 99:99)

Cluster name:        gpfs.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (all rw)


[root at gpfs01 ~]# mmlsconfig
Configuration data for cluster gpfs.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName gpfs.oerc.local
clusterId 748734524680043237
autoload yes
minReleaseLevel 3.4.0.7
dmapiFileHandleSize 32
maxMBpS 6400
maxblocksize 2M
pagepool 4G
[cnfs0,cnfs1,cnfs2,cnfs3]
pagepool 2G
[common]
tiebreakerDisks vd0_0;vd2_2;vd5_5
cnfsSharedRoot /gpfs/.ha
nfsPrefetchStrategy 1
cnfsVIP gpfs-nfs
subnets 10.200.0.0
cnfsMountdPort 4000
cnfsNFSDprocs 128
[common]
adminMode central

File systems in cluster gpfs.oerc.local:
----------------------------------------
/dev/gpfs


## GPFS Cluster 'cpdn.oerc.local' ##

[root at cpdn-ppc01 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         cpdn.oerc.local
  GPFS cluster id:           10699506775530551223
  GPFS UID domain:           cpdn.oerc.local
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    cpdn-ppc02.oerc.local
  Secondary server:  cpdn-ppc03.oerc.local

 Node  Daemon node name       IP address    Admin node name        Designation
-------------------------------------------------------------------------------
   1   cpdn-ppc01.oerc.local  10.100.10.60  cpdn-ppc01.oerc.local  quorum
   2   cpdn-ppc02.oerc.local  10.100.10.61  cpdn-ppc02.oerc.local  quorum-manager
   3   cpdn-ppc03.oerc.local  10.100.10.62  cpdn-ppc03.oerc.local  quorum-manager

[root at cpdn-ppc01 ~]# mmremotecluster show all
Cluster name:    gpfs.oerc.local
Contact nodes:   10.100.10.21,10.100.10.22
SHA digest:      e7a68ff688d6ef055eb40fe74677b272d6c60879
File systems:    gpfs (gpfs)


[root at cpdn-ppc01 ~]# mmauth show all
Cluster name:        gpfs.oerc.local
Cipher list:         AUTHONLY
SHA digest:          e7a68ff688d6ef055eb40fe74677b272d6c60879
File system access:  (none authorized)

Cluster name:        cpdn.oerc.local (this cluster)
Cipher list:         (none specified)
SHA digest:          e9a2dc678a62d6c581de0b89b49a90f28f401327
File system access:  (all rw)


[root at cpdn-ppc01 ~]# mmremotefs show all
Local Name  Remote Name  Cluster name       Mount Point        Mount Options    Automount  Drive  Priority
gpfs        gpfs         gpfs.oerc.local    /gpfs              rw               yes          -        0


[root at cpdn-ppc01 ~]# mmlsconfig
Configuration data for cluster cpdn.oerc.local:
-----------------------------------------------
myNodeConfigNumber 1
clusterName cpdn.oerc.local
clusterId 10699506775530551223
autoload yes
dmapiFileHandleSize 32
minReleaseLevel 3.4.0.7
subnets 10.200.0.0
pagepool 4G
[cpdn-ppc02,cpdn-ppc03]
pagepool 2G
[common]
traceRecycle local
trace all 4 tm 2 thread 1 mutex 1 vnode 2 ksvfs 3 klockl 2 io 3 pgalloc 1 mb 1 lock 2 fsck 3
adminMode central

File systems in cluster cpdn.oerc.local:
----------------------------------------
(none)


As far as I can see I have everything set up and have exchanged the public keys for each cluster and installed them using the -k switch for mmremotecluster and mmauth on the respective clusters. I've tried reconfiguring the admin-interface and daemon-interface names on the cpdn.oerc.local cluster but get the same error (stab in the dark after looking at some trace dumps and seeing IP address inconsistencies). Now I'm worried I've missed something really obvious! Any help greatly appreciated. Here's some trace output from the mmmount gpfs command when run from the cpdn.oerc.local cluster:


35.736808   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) signalling condvar 0x7F8968092D90 (0x7F8968092D90) (ThreadSuspendResumeCondvar) waitCount 1
35.736811   2506 TRACE_MUTEX: internalSignalSave: Created event word 0xFFFF88023AEE1108 for mutex ThreadSuspendResumeMutex
35.736812   2506 TRACE_MUTEX: Releasing mutex 0x1489F28 (0x1489F28) (ThreadSuspendResumeMutex) in daemon (threads waiting)
35.736894   2506 TRACE_BASIC: Wed May  7 08:24:15.991 2014: Waiting to join remote cluster gpfs.oerc.local
35.736927   2506 TRACE_MUTEX: Thread 0x320031 (MountHandlerThread) waiting on condvar 0x14BAB50 (0x14BAB50) (ClusterConfigurationBCHCond): waiting to join remote cluster
35.737369   2643 TRACE_SP: RunProbeCluster: enter. EligibleQuorumNode 0 maxPingIterations 10
35.737371   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 1/10 nToTry 2 nResponses 0 nProbed 0
35.739561   2643 TRACE_DLEASE: Pinger::send: node <c1p2> err 0
35.739620   2643 TRACE_DLEASE: Pinger::send: node <c1p1> err 0
35.739624   2643 TRACE_THREAD: Thread 0x324050 (ProbeRemoteClusterThread) delaying until 1399447456.994516000: waiting for ProbeCluster ping response
35.739726   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p2> 10.100.10.22
35.739728   2579 TRACE_DLEASE: Pinger::receiveLoop: echoreply from <c1p1> 10.100.10.21
35.739730   2579 TRACE_BASIC: cxiRecvfrom: sock 9 buf 0x7F896CB64960 len 128 flags 0 failed with err 11
35.824879   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 leader <c0n1> (me 0) remountRetryNeeded 0
35.824885   2596 TRACE_DLEASE: renewLease: leaseage 10 (100 ticks/sec) now 429499910 lastLeaseReplyReceived 429498823
35.824891   2596 TRACE_TS: tscSend: service 00010001 msg 'ccMsgDiskLease' n_dest 1 data_len 4 msg_id 94 msg 0x7F89500098B0 mr 0x7F89500096E0
35.824894   2596 TRACE_TS: acquireConn enter: addr <c0n1>
35.824895   2596 TRACE_TS: acquireConn exit: err 0 connP 0x7F8948025210
35.824898   2596 TRACE_TS: sendMessage dest <c0n1> 10.200.61.1 cpdn-ppc02: msg_id 94 type 14 tagP 0x7F8950009CB8 seq 89, state initial
35.824957   2596 TRACE_TS: llc_send_msg: returning 0
35.824958   2596 TRACE_TS: tscSend: replies[0] dest <c0n1>, status pending, err 0
35.824960   2596 TRACE_TS: tscSend: rc = 0x0
35.824961   2596 TRACE_DLEASE: checkAndRenewLease: cluster 0 nextLeaseCheck in 2 sec
35.824989   2596 TRACE_THREAD: Thread 0x20C04D (DiskLeaseThread) delaying until 1399447458.079879000: RunLeaseChecks waiting for next check time
35.825509   2642 TRACE_TS: socket_dequeue_next: returns 8
35.825511   2642 TRACE_TS: socket_dequeue_next: returns -1
35.825513   2642 TRACE_TS: receiverEvent enter: sock 8 event 0x5 state reading header
35.825527   2642 TRACE_TS: service_message: enter: msg 'reply', msg_id 94 seq 88 ackseq 89, from <c0n1> 10.200.61.1, active 0
35.825531   2642 TRACE_TS: tscHandleMsgDirectly: service 00010001, msg 'reply', msg_id 94, len 4, from <c0n1> 10.100.10.61
35.825533   2642 TRACE_TS: HandleReply: status success, err 0; 0 msgs pending after this reply
35.825534   2642 TRACE_MUTEX: Acquired mutex 0x7F896805AC68 (0x7F896805AC68) (PendMsgTabMutex) in daemon using trylock
35.825537   2642 TRACE_DLEASE: renewLease: ccMsgDiskLease reply.status 6 err 0 from <c0n1> (expected 10.100.10.61) current leader 10.100.10.61
35.825545   2642 TRACE_DLEASE: DMS timer [0] started, delay 58, time 4295652
35.825546   2642 TRACE_DLEASE: updateMyLease: oldLease 4294988 newLease 4294999 (35 sec left) leaseLost 0
35.825556   2642 TRACE_BASIC: cxiRecv: sock 8 buf 0x7F8954010BE8 len 32 flags 0 failed with err 11
35.825557   2642 TRACE_TS: receiverEvent exit: sock 8 err 54 newTypes 1 state reading header
36.739811   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739814   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p2> gid 00000000:00000000 flags 01
36.739824   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 95 msg 0x7F8950009F20 mr 0x7F8950009D50
36.739829   2643 TRACE_TS: acquireConn enter: addr <c1p2>
36.739831   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F8964025040
36.739835   2643 TRACE_TS: sendMessage dest <c1p2> 10.100.10.22 10.100.10.22: msg_id 95 type 36 tagP 0x7F895000A328 seq 1, state initial
36.739838   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.22 (primary listed 1 0)
36.739914   2643 TRACE_BASIC: Wed May  7 08:24:16.994 2014: Remote mounts are not enabled within this cluster.
36.739963   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.22
36.739965   2643 TRACE_TS: llc_send_msg: returning 693
36.739966   2643 TRACE_TS: tscSend: replies[0] dest <c1p2>, status node_failed, err 693
36.739968   2643 TRACE_MUTEX: Acquired mutex 0x7F896805AC90 (0x7F896805AC90) (PendMsgTabMutex) in daemon using trylock
36.739969   2643 TRACE_TS: tscSend: rc = 0x1
36.739970   2643 TRACE_SP: RunProbeCluster: reply rc 693 tryHere <none>, flags 0
36.739972   2643 TRACE_SP: RunProbeCluster: cl 1 gpnStatus none prevLeaseSeconds 0 loopIteration 1 pingIteration 2/10 nToTry 2 nResponses 2 nProbed 1
36.739973   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739974   2643 TRACE_SP: RunProbeCluster: sending probe 1 to <c1p1> gid 00000000:00000000 flags 01
36.739977   2643 TRACE_TS: tscSend: service 00010001 msg 'ccMsgProbeCluster2' n_dest 1 data_len 100 msg_id 96 msg 0x7F895000A590 mr 0x7F895000A3C0
36.739978   2643 TRACE_TS: acquireConn enter: addr <c1p1>
36.739979   2643 TRACE_TS: acquireConn exit: err 0 connP 0x7F89640258D0
36.739980   2643 TRACE_TS: sendMessage dest <c1p1> 10.100.10.21 10.100.10.21: msg_id 96 type 36 tagP 0x7F895000A998 seq 1, state initial
36.739982   2643 TRACE_TS: llc_pick_dest_addr: use default addrs from 10.100.10.60 to 10.100.10.21 (primary listed 1 0)
36.739993   2643 TRACE_BASIC: Wed May  7 08:24:16.995 2014: Remote mounts are not enabled within this cluster.
36.740003   2643 TRACE_TS: TcpConn::make_connection: status=init, err=720, dest 10.100.10.21
36.740005   2643 TRACE_TS: llc_send_msg: returning 693
36.740005   2643 TRACE_TS: tscSend: replies[0] dest <c1p1>, status node_failed, err 693


Sorry if the formatting above gets horribly screwed. Thanks for any assistance,

Luke

--

Luke Raimbach
IT Manager
Oxford e-Research Centre
7 Keble Road,
Oxford,
OX1 3QG

+44(0)1865 610639


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140507/f289cd64/attachment-0003.htm>

From pavel.pokorny at datera.cz  Fri May  9 10:15:54 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Fri, 9 May 2014 17:15:54 +0800
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
Message-ID: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>

Hello,
our customer would like to deny some of the file extensions (.mp3, .avi,
...) to be stored on GPFS FS. Is there any way to deny some of the
extensions?
Right now we only find the solution to make a placement rule that directs
the files to metadataonly storage pool.

This leads me to second question. Is there any easy way to make a group of
extension (some type of templates) for placement rules? Because there is a
lot of extensions, customer would like to deny, and the placement rules
files start to grow :-(.

Thank you very much for your ideas and answers.
Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140509/b151676f/attachment-0003.htm>

From jonathan at buzzard.me.uk  Fri May  9 14:21:58 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Fri, 09 May 2014 14:21:58 +0100
Subject: [gpfsug-discuss] GPFS Placement rules - group file extension?
In-Reply-To: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
References: <CAHy3nHLbYTC-b83Fw966DCumOKC+2rarFY9xVAnn_A6vc97Ndg@mail.gmail.com>
Message-ID: <1399641718.19065.22.camel@buzzard.phy.strath.ac.uk>

On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> Hello,
> our customer would like to deny some of the file extensions
> (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> some of the extensions?
> Right now we only find the solution to make a placement rule that
> directs the files to metadataonly storage pool.
> 

Create a really small storage pool and fill it in advance.

> 
> This leads me to second question. Is there any easy way to make a
> group of extension (some type of templates) for placement rules?
> Because there is a lot of extensions, customer would like to deny, and
> the placement rules files start to grow :-(.

Note if you deny extensions out right you are more likely to force users
to come up with work arounds. The simple expediency of changing the file
extension for example will lead you to a game of whack-a-mole.

Better to given them really rubbish performance and don't talk about it.
The chances are this is less of a business cost than dealing with the
outright banning, especially when there turns out to be some legitimate
business reason for having multimedia stuff, if even only occasionally.

So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
files. Performance will suck, but the users probably won't realize what
is going on, and are unlikely to come to support asking why their itunes
collection is really slow. That said if you are doing ILM with a
nearline pool holding most of the storage, it is likely to be near idle
most of the time so just force all multimedia stuff to your nearline
pool is adequate.

JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From pavel.pokorny at datera.cz  Mon May 12 03:46:40 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Mon, 12 May 2014 10:46:40 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
Message-ID: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>

Hello,
after discussion with Marc A Kaplan from IBM I finished with this solution.
So I am sharing it with all other user just in case you need it in future.

Example placement policy file:

/* --- Enable .avi so it will not be denied by default --- */
RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$'])

/* --- Deny all following extensions --- */
RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|\.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|\.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|\.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|\.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|\.if$|\.spiff$|\.tif*$'])

/* --- Enable every other file extensions --- */
RULE 'DEFAULT' SET POOL 'data'

Bye, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Sat, May 10, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: GPFS Placement rules - group file extension?
>       (Jonathan Buzzard)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 09 May 2014 14:21:58 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] GPFS Placement rules - group file
>         extension?
> Message-ID: <1399641718.19065.22.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Fri, 2014-05-09 at 17:15 +0800, Pavel Pokorny wrote:
> > Hello,
> > our customer would like to deny some of the file extensions
> > (.mp3, .avi, ...) to be stored on GPFS FS. Is there any way to deny
> > some of the extensions?
> > Right now we only find the solution to make a placement rule that
> > directs the files to metadataonly storage pool.
> >
>
> Create a really small storage pool and fill it in advance.
>
> >
> > This leads me to second question. Is there any easy way to make a
> > group of extension (some type of templates) for placement rules?
> > Because there is a lot of extensions, customer would like to deny, and
> > the placement rules files start to grow :-(.
>
> Note if you deny extensions out right you are more likely to force users
> to come up with work arounds. The simple expediency of changing the file
> extension for example will lead you to a game of whack-a-mole.
>
> Better to given them really rubbish performance and don't talk about it.
> The chances are this is less of a business cost than dealing with the
> outright banning, especially when there turns out to be some legitimate
> business reason for having multimedia stuff, if even only occasionally.
>
> So a RAID1 of a couple of large nearline SAS disks for all MP3 etc.
> files. Performance will suck, but the users probably won't realize what
> is going on, and are unlikely to come to support asking why their itunes
> collection is really slow. That said if you are doing ILM with a
> nearline pool holding most of the storage, it is likely to be near idle
> most of the time so just force all multimedia stuff to your nearline
> pool is adequate.
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 6
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140512/84920933/attachment-0003.htm>

From jonathan at buzzard.me.uk  Mon May 12 12:12:19 2014
From: jonathan at buzzard.me.uk (Jonathan Buzzard)
Date: Mon, 12 May 2014 12:12:19 +0100
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
Message-ID: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>

On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> Hello,
> after discussion with Marc A Kaplan from IBM I finished with this
> solution. So I am sharing it with all other user just in case you need
> it in future.
> 
> 
> Example placement policy file:
> 
> 
> /* --- Enable .avi so it will not be denied by default --- */
> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$'])
> 
> 
> /* --- Deny all following extensions --- */
> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> \.if$|\.spiff$|\.tif*$'])
> 

To demonstrate why this is a wack-a-mole exercise, as a user of your
system I will just save in iPod friendly format, all my audio AAC
encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
audiobooks in .m4b format :-)

I would further note that it is rather trivial to pick a random
extension say .sdf and associate that with just about any program, most
of which examine the contents of the file and don't rely on the
extension.

Further noting if you are continuing in your wack-a-mole exercise I
could use .divx which you have conveniently not blocked.

Finally being the resourceful sort I will just embed it all in a PDF, or
word document, or powerpoint, or ...

I don't know exactly what your customer is expecting, but this sounds
like a management idea, with little understanding of how things work in
the real world.


JAB.

-- 
Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
Fife, United Kingdom.




From zgiles at gmail.com  Mon May 12 15:02:04 2014
From: zgiles at gmail.com (Zachary Giles)
Date: Mon, 12 May 2014 10:02:04 -0400
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
In-Reply-To: <1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
References: <mailman.1.1399719601.770.gpfsug-discuss@gpfsug.org>
	<CAHy3nHK1Z-v1bQ8NpDp5Yq9Qke4BtvipK2k43mGAED920HjniQ@mail.gmail.com>
	<1399893139.27622.11.camel@buzzard.phy.strath.ac.uk>
Message-ID: <CAMYZk=f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg@mail.gmail.com>

To further the wack-a-mole problem:
If you set those extensions to be in the system pool and assume the
system pool has no data-disks, it is still possible for GPFS 3.5 to
create a file in the system pool if one is utilizing the
small-files-in-metadata feature (whatever it's called now) and if the
file is 0-3.9K in size where it will be stored in the xattrs.
So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
if, say, you wanted to restrict .txt files, which are often that
small.



On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
<jonathan at buzzard.me.uk> wrote:
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
>> Hello,
>> after discussion with Marc A Kaplan from IBM I finished with this
>> solution. So I am sharing it with all other user just in case you need
>> it in future.
>>
>>
>> Example placement policy file:
>>
>>
>> /* --- Enable .avi so it will not be denied by default --- */
>> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$'])
>>
>>
>> /* --- Deny all following extensions --- */
>> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
>> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
>> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
>> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
>> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
>> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
>> \.if$|\.spiff$|\.tif*$'])
>>
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss



-- 
Zach Giles
zgiles at gmail.com


From pavel.pokorny at datera.cz  Tue May 13 15:35:25 2014
From: pavel.pokorny at datera.cz (Pavel Pokorny)
Date: Tue, 13 May 2014 22:35:25 +0800
Subject: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28,
	Issue 8 (Pavel Pokorny)
Message-ID: <CAHy3nHL4t0vAC8c7PuFwsa6Wo-T3VCTNSpFc0D=-G9Su4MwrXw@mail.gmail.com>

Hello,
just to clarify. Customer need is very easy: Deny storing selected files
extension in specific filesets.
I also don't like the way to direct placement to metadata only pool. I also
know that small filse can actually be stored there.
I will appreciate more help how to make it. I don't think that expalanation
to customer that his need is stupid, is a good approach to the customer.
If there is or not .divx extension is not important, we care about GPFS
stuff and the extensions were selected by customer.
Thank you, Pavel

--
Ing. Pavel Pokorn?
DATERA s.r.o. | Ovocn? trh 580/2 | Praha | Czech Republic
www.datera.cz | Mobil: +420 602 357 194 | E-mail: pavel.pokorny at datera.cz



On Tue, May 13, 2014 at 7:00 PM, <gpfsug-discuss-request at gpfsug.org> wrote:

> Send gpfsug-discuss mailing list submissions to
>         gpfsug-discuss at gpfsug.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> or, via email, send a message with subject or body 'help' to
>         gpfsug-discuss-request at gpfsug.org
>
> You can reach the person managing the list at
>         gpfsug-discuss-owner at gpfsug.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of gpfsug-discuss digest..."
>
>
> Today's Topics:
>
>    1. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Jonathan Buzzard)
>    2. Re: gpfsug-discuss Digest, Vol 28, Issue 6 (Zachary Giles)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 12 May 2014 12:12:19 +0100
> From: Jonathan Buzzard <jonathan at buzzard.me.uk>
> To: gpfsug-discuss at gpfsug.org
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID: <1399893139.27622.11.camel at buzzard.phy.strath.ac.uk>
> Content-Type: text/plain; charset="UTF-8"
>
> On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> > Hello,
> > after discussion with Marc A Kaplan from IBM I finished with this
> > solution. So I am sharing it with all other user just in case you need
> > it in future.
> >
> >
> > Example placement policy file:
> >
> >
> > /* --- Enable .avi so it will not be denied by default --- */
> > RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$'])
> >
> >
> > /* --- Deny all following extensions --- */
> > RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> > RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> > \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> > \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> > \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> > \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> > \.if$|\.spiff$|\.tif*$'])
> >
>
> To demonstrate why this is a wack-a-mole exercise, as a user of your
> system I will just save in iPod friendly format, all my audio AAC
> encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> audiobooks in .m4b format :-)
>
> I would further note that it is rather trivial to pick a random
> extension say .sdf and associate that with just about any program, most
> of which examine the contents of the file and don't rely on the
> extension.
>
> Further noting if you are continuing in your wack-a-mole exercise I
> could use .divx which you have conveniently not blocked.
>
> Finally being the resourceful sort I will just embed it all in a PDF, or
> word document, or powerpoint, or ...
>
> I don't know exactly what your customer is expecting, but this sounds
> like a management idea, with little understanding of how things work in
> the real world.
>
>
> JAB.
>
> --
> Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> Fife, United Kingdom.
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Mon, 12 May 2014 10:02:04 -0400
> From: Zachary Giles <zgiles at gmail.com>
> To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
> Subject: Re: [gpfsug-discuss] gpfsug-discuss Digest, Vol 28, Issue 6
> Message-ID:
>         <CAMYZk=
> f5aQYqtm_hGJbc7EBqcKJXPi_RPynP6PcwEnUUN0z8Tg at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> To further the wack-a-mole problem:
> If you set those extensions to be in the system pool and assume the
> system pool has no data-disks, it is still possible for GPFS 3.5 to
> create a file in the system pool if one is utilizing the
> small-files-in-metadata feature (whatever it's called now) and if the
> file is 0-3.9K in size where it will be stored in the xattrs.
> So, this only works for mp3's >3.9K :) Probably OK for mp3's but not
> if, say, you wanted to restrict .txt files, which are often that
> small.
>
>
>
> On Mon, May 12, 2014 at 7:12 AM, Jonathan Buzzard
> <jonathan at buzzard.me.uk> wrote:
> > On Mon, 2014-05-12 at 10:46 +0800, Pavel Pokorny wrote:
> >> Hello,
> >> after discussion with Marc A Kaplan from IBM I finished with this
> >> solution. So I am sharing it with all other user just in case you need
> >> it in future.
> >>
> >>
> >> Example placement policy file:
> >>
> >>
> >> /* --- Enable .avi so it will not be denied by default --- */
> >> RULE 'enable_avi' SET POOL 'data' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$'])
> >>
> >>
> >> /* --- Deny all following extensions --- */
> >> RULE 'deny_files' SET POOL 'system' FOR FILESET (nfsexport) WHERE
> >> RegEx(lower(name),['\.avi$|\.mp[ae1234]$|\.mpe?g[23]?$|\.eml$|\.idx$|
> >> \.mbo?x$|\.msg$|\.ost$|\.otf$|\.pab$|\.pst$|\.aac$|\.aif*$|\.as[fx]$|
> >> \.au$|\.flac$|\.m3u$|\.midi?$|\.mov$|\.ogg$|\.qtw?$|\.ram$|\.rmi?$|
> >> \.rmvb$|\.snd$|\.swf$|\.vob$|\.wa[vx]$|\.wm[av]$|\.wvx$|\.bmp$|\.dib$|
> >> \.eps$|\.gif$|\.img$|\.jfif$|\.jpe?g?$|\.pcx$|\.png$|\.psd?$|\.raw$|
> >> \.if$|\.spiff$|\.tif*$'])
> >>
> >
> > To demonstrate why this is a wack-a-mole exercise, as a user of your
> > system I will just save in iPod friendly format, all my audio AAC
> > encoded in .m4a files, all my video H.264 encoded in .m4v files, and my
> > audiobooks in .m4b format :-)
> >
> > I would further note that it is rather trivial to pick a random
> > extension say .sdf and associate that with just about any program, most
> > of which examine the contents of the file and don't rely on the
> > extension.
> >
> > Further noting if you are continuing in your wack-a-mole exercise I
> > could use .divx which you have conveniently not blocked.
> >
> > Finally being the resourceful sort I will just embed it all in a PDF, or
> > word document, or powerpoint, or ...
> >
> > I don't know exactly what your customer is expecting, but this sounds
> > like a management idea, with little understanding of how things work in
> > the real world.
> >
> >
> > JAB.
> >
> > --
> > Jonathan A. Buzzard                 Email: jonathan (at) buzzard.me.uk
> > Fife, United Kingdom.
> >
> >
> > _______________________________________________
> > gpfsug-discuss mailing list
> > gpfsug-discuss at gpfsug.org
> > http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> --
> Zach Giles
> zgiles at gmail.com
>
>
> ------------------------------
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
> End of gpfsug-discuss Digest, Vol 28, Issue 8
> *********************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140513/a9784129/attachment-0003.htm>

From sabujp at gmail.com  Thu May 15 13:48:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 15 May 2014 07:48:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
Message-ID: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>

Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager
automatically reboots (and sometimes fails to automatically reboot) after
mmdelsnapshot is called, either from the filesystem manager itself or from
some other nsd/node . It didn't start happening immediately after we
updated to 17, but we never had this issue when we were at 3.5.0.11 . The
error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.
mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
mmdelsnapshot: Command failed.  Examine previous error messages to
determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup
- CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to
not reboot itself (which is actually worse), but the former does.


It also causes some havoc with CNFS file locking even after the
filesystem manager is rebooted and has come up :


May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
Address already in use


May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known


Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior
or know if it's fixed in 18?


Thanks,

Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140515/f6db747b/attachment-0003.htm>

From sabujp at gmail.com  Mon May 19 16:16:14 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Mon, 19 May 2014 10:16:14 -0500
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/09e7cd91/attachment-0003.htm>

From TOMP at il.ibm.com  Mon May 19 16:35:32 2014
From: TOMP at il.ibm.com (Tomer Perry)
Date: Mon, 19 May 2014 18:35:32 +0300
Subject: [gpfsug-discuss] gpfs 4.1 features?
In-Reply-To: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
References: <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q@mail.gmail.com>
Message-ID: <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702@il.ibm.com>

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm

Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140519/8d477962/attachment-0003.htm>

From johnlang at us.ibm.com  Tue May 20 13:00:19 2014
From: johnlang at us.ibm.com (John Langlois)
Date: Tue, 20 May 2014 08:00:19 -0400
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
Message-ID: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>

Folks,

We're going through a naming transition and you will see some GPFS 
features now described under the codename Elastic Storage.  Don't let the 
codename confuse you ... I'll let you all know the moment we get the final 
Family and Product naming straight. We are changing the name to reflect 
the fact that IBM is significantly increasing investment in this 
technology area and we will stretch GPFS out into the Cloud with Open 
Stack object support (e.g. SWIFT object store guide coming in June).  In 
addition, GPFS will be available in the Softlayer Cloud Managed Service 
Offering catalog soon.  You will be seeing a lot of new and exciting news 
on our roadmaps.

New to GPFS 4.1 features include:
Native encryption and secure cryptography erase (data wiping) that is NIST 
Compliant and FIPS certified.
GPFS client-side IBM Elastic Storage Flash caches that increase I/O 
performance up to 6x
Active File Manager now supports parallel data transfers between sites, 
using multiple gateway nodes 
AFM now supports the GPFS protocol in addition to NFS protocol for file 
transfer
Much simpler socket based pricing - no more PVUs. 
Lots of Usability improvements that include
  a.  NFS Data Migration - Eases data transfer from NFS when upgrading 
hardware or buying a new system 
  b   AFM Optimization  - Improved prefetch performance; support for the 
native GPFS protocol   (described above)
  c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool; 
Backup utility enhancements
  d.  FPO Improvements - Data locality aware file system recovery and 
improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a 
Cinder driver that gives architects access to GPFS features and 
capabilities.

Here's the current external website landing page:  
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager

Phone: 1-919-267-6767
E-mail: johnlang at us.ibm.com
Find me on:   and within IBM on:  






From:   gpfsug-discuss-request at gpfsug.org
To:     gpfsug-discuss at gpfsug.org
Date:   05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                 gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                 http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                 gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                 gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

   1. gpfs 4.1 features? (Sabuj Pattanayek)
   2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:
 <OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>, 
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 . 
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0009.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0010.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0011.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/19a4dcd0/attachment-0003.gif>

From sjhoward at iu.edu  Tue May 20 15:17:32 2014
From: sjhoward at iu.edu (Howard, Stewart Jameson)
Date: Tue, 20 May 2014 14:17:32 +0000
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
Message-ID: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>

Hi All,

My name is Stewart Howard and I work for Indiana University as an admin on a two-site replicated GPFS cluster.  I'm a new member of this mailing list and this is my first post  :)

Recently, we've discovered that small-file performance on our system is pretty lack-luster.  For comparison, here are some numbers:

1)  When transferring large files (~2 GB), we get outstanding performance and can typically saturate the client's network connection.  We generally see about 490 MB/s over a 10Gb line, which should be about right, given that we lose half of our bandwidth to replication.

2)  When transferring a large number of small files, we get a very poor transfer rate, generally on the order of 2 MB/s, writing from a client node *inside* the GPFS cluster.

I'm wondering if anyone else has experience with similar performance issues and what ended up being the cause/solution.  Also, I would be interested in hearing any general rules-of-thumb that the group has found helpful in balancing performance between large-file and small-file I/O.

We have gathered some diagnostic information while performing various small-file I/O operations, as well as a variety of metadata operations in quick succession.  I'd be happy to share results of the diagnostics, if it would help provide context.

Thank you so much for all of your help!

Stewart Howard
Indiana University


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/63b40754/attachment-0003.htm>

From dhildeb at us.ibm.com  Tue May 20 18:29:51 2014
From: dhildeb at us.ibm.com (Dean Hildebrand)
Date: Tue, 20 May 2014 10:29:51 -0700
Subject: [gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
In-Reply-To: <OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
References: <mailman.1.1400583601.22418.gpfsug-discuss@gpfsug.org>
	<OF54E6FCAF.16A20175-ON87257CDE.003E0D78-85257CDE.0041F612@us.ibm.com>
Message-ID: <OF2F748B64.B172A915-ON88257CDE.00601714-88257CDE.00601DF1@us.ibm.com>


Nice email!

Dean Hildebrand
IBM Master Inventor and Manager | Scale-out Storage Software
IBM Almaden Research Center




From:	John Langlois/Raleigh/IBM at IBMUS
To:	gpfsug-discuss at gpfsug.org
Date:	05/20/2014 05:01 AM
Subject:	[gpfsug-discuss] ANSWER:  GPFS 4.1 New Features
Sent by:	gpfsug-discuss-bounces at gpfsug.org



Folks,

We're going through a naming transition and you will see some GPFS features
now described under the codename Elastic Storage.  Don't let the codename
confuse you ... I'll let you all know the moment we get the final Family
and Product naming straight. We are changing the name to reflect the fact
that IBM is significantly increasing investment in this technology area and
we will stretch GPFS out into the Cloud with Open Stack object support
(e.g. SWIFT object store guide coming in June).  In addition, GPFS will be
available in the Softlayer Cloud Managed Service Offering catalog soon.
You will be seeing a lot of new and exciting news on our roadmaps.

New to GPFS 4.1 features include:
   1.	Native encryption and secure cryptography erase (data wiping) that is
      NIST Compliant and FIPS certified.
   2.	GPFS client-side IBM Elastic Storage Flash caches that increase I/O
      performance up to 6x
   3.	Active File Manager now supports parallel data transfers between
      sites, using multiple gateway nodes
   4.	AFM now supports the GPFS protocol in addition to NFS protocol for
      file transfer
   5.	Much simpler socket based pricing - no more PVUs.
   6.	Lots of Usability improvements that include
       a.  NFS Data Migration - Eases data transfer from NFS when upgrading
      hardware or buying a new system
       b   AFM Optimization  - Improved prefetch performance; support for
      the native GPFS protocol   (described above)
       c.  Backup/restore Enhancements - New Fileset Snapshot Restore tool;
      Backup utility enhancements
       d.  FPO Improvements - Data locality aware file system recovery and
      improved performance

Not new, but included here as a reminder:  OpenStack Havana includes a
Cinder driver that gives architects access to GPFS features and
capabilities.

Here's the current external website landing page:
http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/gpfs/



Regards,

John Langlois
IBM Elastic Storage Product Manager
                                                                                                                                      
                                                                                                                                      
                                                                                                                                      
 Phone: 1-919-267-6767                                                                                                            IBM 
 E-mail: johnlang at us.ibm.com                                                                                                          
 Find me on: LinkedIn:                                                                                                                
 http://www.linkedin.com/profile/view?id=24557762&authType=NAME_SEARCH&authToken=b9zY&locale=en_US&sr                                 
 Twitter: https://twitter.com/johnlang123 and within IBM on: IBM Connections:                                                         
 https://w3-connections.ibm.com/profiles/html/profileView.do?userid=69bf0ec0-8f0a-1028-94ba-db07163b5                                 
                                                                                                                                      
                                                                                                                                      






From:        gpfsug-discuss-request at gpfsug.org
To:        gpfsug-discuss at gpfsug.org
Date:        05/20/2014 07:00 AM
Subject:        gpfsug-discuss Digest, Vol 28, Issue 11
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Send gpfsug-discuss mailing list submissions to
                gpfsug-discuss at gpfsug.org

To subscribe or unsubscribe via the World Wide Web, visit
                http://gpfsug.org/mailman/listinfo/gpfsug-discuss
or, via email, send a message with subject or body 'help' to
                gpfsug-discuss-request at gpfsug.org

You can reach the person managing the list at
                gpfsug-discuss-owner at gpfsug.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of gpfsug-discuss digest..."


Today's Topics:

  1. gpfs 4.1 features? (Sabuj Pattanayek)
  2. Re: gpfs 4.1 features? (Tomer Perry)


----------------------------------------------------------------------

Message: 1
Date: Mon, 19 May 2014 10:16:14 -0500
From: Sabuj Pattanayek <sabujp at gmail.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<CAEeMGHvSdeSePoGYV5XJ_z5PKBpU7eFX6pmPV3aEsM1MjDhE9Q at mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/09e7cd91/attachment-0001.html
>

------------------------------

Message: 2
Date: Mon, 19 May 2014 18:35:32 +0300
From: Tomer Perry <TOMP at il.ibm.com>
To: gpfsug main discussion list <gpfsug-discuss at gpfsug.org>
Subject: Re: [gpfsug-discuss] gpfs 4.1 features?
Message-ID:

<OF6E10D0A7.5DC41F9C-ONC2257CDD.00559912-C2257CDD.0055A702 at il.ibm.com>
Content-Type: text/plain; charset="us-ascii"

I guess you can check the "what's new" on the GPFS docs:
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/topic/com.ibm.cluster.gpfs.v4r1.gpfs100.doc/bl1xx_soc.htm


Tomer.




From:   Sabuj Pattanayek <sabujp at gmail.com>
To:     gpfsug main discussion list <gpfsug-discuss at gpfsug.org>,
Date:   05/19/2014 06:16 PM
Subject:        [gpfsug-discuss] gpfs 4.1 features?
Sent by:        gpfsug-discuss-bounces at gpfsug.org



I can't seem to find a page that lists features that are new to 4.1 .
Anyone have a link to such a page/presentation, etc?

Thanks,
Sabuj_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://gpfsug.org/pipermail/gpfsug-discuss/attachments/20140519/8d477962/attachment-0001.html
>

------------------------------

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


End of gpfsug-discuss Digest, Vol 28, Issue 11
**********************************************

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0006.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62923814.jpg
Type: image/jpeg
Size: 518 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0009.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62235135.jpg
Type: image/jpeg
Size: 638 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0010.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62861266.jpg
Type: image/jpeg
Size: 1208 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0011.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 62781961.gif
Type: image/gif
Size: 1851 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140520/3fe10d1c/attachment-0007.gif>

From chekh at stanford.edu  Wed May 21 21:04:52 2014
From: chekh at stanford.edu (Alex Chekholko)
Date: Wed, 21 May 2014 13:04:52 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
Message-ID: <537D06E4.3090603@stanford.edu>

Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860


From oehmes at us.ibm.com  Wed May 21 22:42:51 2014
From: oehmes at us.ibm.com (Sven Oehme)
Date: Wed, 21 May 2014 14:42:51 -0700
Subject: [gpfsug-discuss] Mitigating Poor Small-file I/O Performance
In-Reply-To: <537D06E4.3090603@stanford.edu>
References: <8D694C60498A2942A883CA5388A911562978741E@IU-MSSG-MBX107.ads.iu.edu>
	<537D06E4.3090603@stanford.edu>
Message-ID: <OF11EC8E07.09F9EDEE-ON88257CDF.0074EF02-88257CDF.007747E9@us.ibm.com>

Hi Alex, Stewart,

the problem is more likely a latency and lack of parallelism of the 
transfer.
 
if you have 1 thread that transfers files of 1k in size you don't get a 
lot of bandwidth as it transfers one at a time and the latency kills the 
bandwidth. to explain this on an example :

assume your network between client and server is 10 gigabit and both nodes 
are capable of pushing this.

if it takes 1 ms for reading + writing each 1 kb file you get ~ 1.02 
MB/sec
if your filesize changes to 4k, even you don't do anything else it goes up 
to 4.06 MB/sec
if you can reduce latency on read or write to lets say 100us the same 
process would transfer 37.86 MB/sec

so this is a latency / parallelism problem and this has nothing to do with 
GPFS, you would have exactly the same issue.
the solution is to copy the data with a tool that does it multi threaded 
as the numbers above are based on 1 thread. 
if you would have 100us read+write time and transfer 4k files with 10 
threads in parallel the same transfer would be close to 400 MB/sec.

hope that helps. Sven 
------------------------------------------
Sven Oehme 
Scalable Storage Research 
email: oehmes at us.ibm.com 
Phone: +1 (408) 824-8904 
IBM Almaden Research Lab 
------------------------------------------



From:   Alex Chekholko <chekh at stanford.edu>
To:     gpfsug-discuss at gpfsug.org
Date:   05/21/2014 01:05 PM
Subject:        Re: [gpfsug-discuss] Mitigating Poor Small-file I/O 
Performance
Sent by:        gpfsug-discuss-bounces at gpfsug.org



Hi Stewart,

First, a good simple reproducible benchmark is fdtree:
https://computing.llnl.gov/?set=code&page=sio_downloads

Something simple like this should take a min or two:
   bash fdtree.bash  -l 3 -s 64

Or the same exact small run can take up to hours on a slow system.

For GPFS, since it's a clustered filesystem, first you need to make sure 
you're looking at the aggregate performance and not just on one client. 
  Perhaps your filesystem is performing great, but it's maxed out at 
that moment when you run your test from your single client.  So you need 
to be able to monitor the disk system.

In general, the answer to your question is, in order of simplicity: add 
more spindles, possibly also separate the metadata out to separate 
storage, possibly make your filesystem block size smaller.

The first you can do by adding more hardware, the second is easier when 
you design your whole system, though possible to do on a running 
filesystem.  The third can only be done at filesystem creation.

For "small files", how "small" is "small".  I guess generally we mean 
smaller than filesystem block size.

Regards,
Alex


On 5/20/14, 7:17 AM, Howard, Stewart Jameson wrote:
> Hi All,
>
> My name is Stewart Howard and I work for Indiana University as an admin
> on a two-site replicated GPFS cluster.  I'm a new member of this mailing
> list and this is my first post  :)
>
> Recently, we've discovered that small-file performance on our system is
> pretty lack-luster.  For comparison, here are some numbers:
>
> 1)  When transferring large files (~2 GB), we get outstanding
> performance and can typically saturate the client's network connection.
> We generally see about 490 MB/s over a 10Gb line, which should be about
> right, given that we lose half of our bandwidth to replication.
>
> 2)  When transferring a large number of small files, we get a very poor
> transfer rate, generally on the order of 2 MB/s, writing from a client
> node *inside* the GPFS cluster.
>
> I'm wondering if anyone else has experience with similar performance
> issues and what ended up being the cause/solution.  Also, I would be
> interested in hearing any general rules-of-thumb that the group has
> found helpful in balancing performance between large-file and small-file
> I/O.
>
> We have gathered some diagnostic information while performing various
> small-file I/O operations, as well as a variety of metadata operations
> in quick succession.  I'd be happy to share results of the diagnostics,
> if it would help provide context.
>
> Thank you so much for all of your help!
>
> Stewart Howard
> Indiana University
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at gpfsug.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
chekh at stanford.edu 347-401-4860
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at gpfsug.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140521/08d101d4/attachment-0003.htm>

From sabujp at gmail.com  Tue May 27 22:29:02 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Tue, 27 May 2014 16:29:02 -0500
Subject: [gpfsug-discuss] feature request : ability to run multiple snapshot
	related commands simultaneously
Message-ID: <CAEeMGHuFnKtH1a8eirKqu5r5wxeOfC_wVahpwSnPio8k8XV51w@mail.gmail.com>

Sometimes snapshot deletions can take a very long time. While they're
running no other snapshot related commands can be run, e.g. it doesn't look
like I can run an mmcrsnapshot on the same fileset while a global snapshot
deletion or a snapshot deletion on the same fileset is ongoing.

Today's snapshot deletion on a global snapshot created about 19 hours ago
is going painfully slowly for some reason. It's also affecting CNFS and SMB
performance with periods of hangs for about 30+ seconds. This is not the
norm on our fs. GPFS access continues to be ok during these NFS/SMB hangs.
Another problem it causes is that snapshots on the root fileset cannot be
created while this global snapshot is being deleted. This disrupts our
daily snapshot creation schedule and I also have to switch to running
mmbackup without a snapshot, since I won't be able to create a new global
snapshot if another global snapshot is being deleted.

If possible I'd request that running multiple snapshot related commands
simultaneously be allowed, at least the ability to run an mmcrsnapshot
while an mmdelsnapshot is running would be nice. I can see that running
multiple global snapshot deletions or multiple simultaneous snapshot
deletions on the same fileset would be risky/very difficult (or impossible)
to implement but it seems like it should be possible to "quiesce" an
mmdelsnapshot for a few seconds so that an mmcrsnapshot on the same fileset
can be completed. This can be done manually by interrupting an ongoing
snapshot deletion, creating the new snapshot, and continuing the deletion
but would be nice if this could be fully automated . What would be even
cooler would be a snapshot command queuing system for deletion requests
within the same fileset, that way snapshot deletions would be queued up and
automatically occur as soon as deletions in the same fileset are complete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140527/edbbdf15/attachment-0003.htm>

From sabujp at gmail.com  Fri May 30 02:34:00 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Thu, 29 May 2014 20:34:00 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
Message-ID: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>

This is still happening in 3.5.0.18 and when a snapshot is being deleted it
slows NFS read speeds to a crawl (but not gpfs and not NFS writes).


On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com> wrote:

> Hi all,
>
> We're running 3.5.0.17 now and it looks like the filesystem manager
> automatically reboots (and sometimes fails to automatically reboot) after
> mmdelsnapshot is called, either from the filesystem manager itself or from
> some other nsd/node . It didn't start happening immediately after we
> updated to 17, but we never had this issue when we were at 3.5.0.11 . The
> error mmdelsnapshot throws at some point is :
>
> Lost connection to file system daemon.
> mmdelsnapshot: An internode connection between GPFS nodes was disrupted.
> mmdelsnapshot: Command failed.  Examine previous error messages to
> determine cause.
>
> It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.
>
>
> It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :
>
>
> May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:
> Address already in use
>
>
> May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port
>
> for RPC socket: Name or service not known
>
>
> Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?
>
>
> Thanks,
>
> Sabuj
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140529/76d1258b/attachment-0003.htm>

From bbanister at jumptrading.com  Fri May 30 14:35:55 2014
From: bbanister at jumptrading.com (Bryan Banister)
Date: Fri, 30 May 2014 13:35:55 +0000
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
Message-ID: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>

This sounds like a serious problem and you should open a PMR with IBM to get direct guidance.

I normally will take a GPFS trace during a problem like this from all of the nodes that are affected or directly involved in the operation.

Hope that helps,
-Bryan

From: gpfsug-discuss-bounces at gpfsug.org [mailto:gpfsug-discuss-bounces at gpfsug.org] On Behalf Of Sabuj Pattanayek
Sent: Thursday, May 29, 2014 8:34 PM
To: gpfsug main discussion list
Subject: Re: [gpfsug-discuss] filesystem manager crashes every time mmdelsnapshot (from either the filesystem manager or some other nsd/client) is called

This is still happening in 3.5.0.18 and when a snapshot is being deleted it slows NFS read speeds to a crawl (but not gpfs and not NFS writes).

On Thu, May 15, 2014 at 7:48 AM, Sabuj Pattanayek <sabujp at gmail.com<mailto:sabujp at gmail.com>> wrote:
Hi all,

We're running 3.5.0.17 now and it looks like the filesystem manager automatically reboots (and sometimes fails to automatically reboot) after mmdelsnapshot is called, either from the filesystem manager itself or from some other nsd/node . It didn't start happening immediately after we updated to 17, but we never had this issue when we were at 3.5.0.11 . The error mmdelsnapshot throws at some point is :

Lost connection to file system daemon.

mmdelsnapshot: An internode connection between GPFS nodes was disrupted.

mmdelsnapshot: Command failed.  Examine previous error messages to

determine cause.

It also causes an mmfs generic error and or a kernel: BUG: soft lockup - CPU#15 stuck for 67s! [mmfsd:39266], the latter causes the system to not reboot itself (which is actually worse), but the former does.



It also causes some havoc with CNFS file locking even after the filesystem manager is rebooted and has come up :



May 15 07:10:12 mako-nsd1 sm-notify[19387]: Failed to bind RPC socket:

Address already in use



May 15 07:21:03 mako-nsd1 sm-notify[11052]: Invalid bind address or port

for RPC socket: Name or service not known



Saw some snapshot related fixes in 3.5.0.18, anyone seen this behavior or know if it's fixed in 18?



Thanks,

Sabuj



________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/d06dcb69/attachment-0003.htm>

From sabujp at gmail.com  Fri May 30 14:49:15 2014
From: sabujp at gmail.com (Sabuj Pattanayek)
Date: Fri, 30 May 2014 08:49:15 -0500
Subject: [gpfsug-discuss] filesystem manager crashes every time
 mmdelsnapshot (from either the filesystem manager or some other nsd/client)
 is called
In-Reply-To: <21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
References: <CAEeMGHtab+ig+rEPubFvcw6vdyXZmxj41RKUvgROc5zvqW7PvQ@mail.gmail.com>
	<CAEeMGHtxusPyjQtjx3FJvv+y-HLVTk=Hgk0y0HpZG09DoVukcw@mail.gmail.com>
	<21BC488F0AEA2245B2C3E83FC0B33DBB725D53@CHI-EXCHANGEW1.w2k.jumptrading.com>
Message-ID: <CAEeMGHuGfMrVPYd97wPQ_FfCj=ocEzFb7fP9Rx0zQC623==b=A@mail.gmail.com>

On Fri, May 30, 2014 at 8:35 AM, Bryan Banister <bbanister at jumptrading.com>
wrote:

>  This sounds like a serious problem and you should open a PMR with IBM to
> get direct guidance.
>
>
>
> I normally will take a GPFS trace during a problem like this from all of
> the nodes that are affected or directly involved in the operation.
>

easier said than done, I've tried at least 10 times. It's almost as if the
act of running the mmtrace prevents the error from occurring, we've sent
internaldumps and gpfs.snap that were generated to ddn. Not having much
luck in figuring out if it's been sent through to IBM yet or not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20140530/dd6729bd/attachment-0003.htm>