[gpfsug-discuss] python package installs fail due to attribute copy error (GPFS 5.0.2)

[mark.bergman at uphs.upenn.edu]

We're running GPFS 5.0.2 (DSS-G 2.2a, in the process of upgrading to
2.4b/GPFS 5.0.3.2) with NFSv4 ACLs and no explicit ACLs on the root or
dependent filesets.

While installing python packages within a python virtual env stored under
GPFS, we get failures with "permission denied" on the destination while
trying to copy attributes. The problem happens to root & end-users,
and is not related to the directory permissions, ownership, or group,
and is consistent and repeatable.

For example, trying to use 'conda' to create a virtual environment results in the following message (severely trimmed):

	# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
	
	    Traceback (most recent call last):
	
	      in clone_env
	        shutil.copystat(src, dst)
	      in copystat
	        _copyxattr(src, dst, follow_symlinks=follow)
	      in _copyxattr
	        os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
	    PermissionError: [Errno 13] Permission denied: '/gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6/lib/python3.5/site-packages/dm/neuralnet/__pycache__/__init__.cpython-35.pyc'
	
	`$ python/anaconda/3/bin/conda create --prefix /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6 --clone /gpfs/fs1/rootdir/external/python/anaconda/3/envs/py3.5.6`


Note that the destination directory path is created, and the destination file (__init__.cython-35.pyc) is created...the exception is thrown when python
tries to apply the ACLs from the source to the destination.

The problem seems to be simiar to this:
        https://github.com/conda/conda/issues/5251
but our version of create.py is newer and already has the suggested fix.
       
or similar to this:
	https://bugs.python.org/issue24538
that's affecting panfs.

We've got a work-around using symlinks, but it's not something for average users.

Has anyone seen this issue?

Thanks,

Mark "about to post to various python fora" Bergman