[gpfsug-discuss] Those users.... millions of files per directory - not necessarily a mistake

[Bryan Banister]

Thankfully all application developers completely understand why listing directories are a bad idea... ;o)

Or at least they will learn the hard way otherwise,
-B

From: gpfsug-discuss-bounces at spectrumscale.org <gpfsug-discuss-bounces at spectrumscale.org> On Behalf Of Marc A Kaplan
Sent: Thursday, August 23, 2018 12:01 PM
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Subject: Re: [gpfsug-discuss] Those users.... millions of files per directory - not necessarily a mistake

Note: External Email
________________________________
Even with nfs or samba export you're probably okay as long as the application does not attempt to list the directory.  Just probe it with stat/open/create/unlink.




From:        david_johnson at brown.edu<mailto:david_johnson at brown.edu>
To:        gpfsug main discussion list <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
Date:        08/23/2018 11:34 AM
Subject:        Re: [gpfsug-discuss] Those users.... millions of files per        directory - not necessarily a mistake
Sent by:        gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org>
________________________________



But heaven help you if you export the gpfs on nfs or cifs.

  -- ddj
Dave Johnson

On Aug 23, 2018, at 11:23 AM, Marc A Kaplan <makaplan at us.ibm.com<mailto:makaplan at us.ibm.com>> wrote:

Millions of files per directory, may well be a mistake...

BUT there are some very smart use cases that might take advantage of GPFS having good performance with large directories --
because GPFS uses extensible hashing -- it is better to store millions of files in a single GPFS directory than artificially scatter them among directories based on the mistaken notion that large directories are bad.  (Yeah, they are in most implementations, but not in GPFS.)

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss



________________________________

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential, or privileged information and/or personal data. If you are not the intended recipient, you are hereby notified that any review, dissemination, or copying of this email is strictly prohibited, and requested to notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request, or solicitation of any kind to buy, sell, subscribe, redeem, or perform any type of transaction of a financial product. Personal data, as defined by applicable data privacy laws, contained in this email may be processed by the Company, and any of its affiliated or related companies, for potential ongoing compliance and/or business-related purposes. You may have rights regarding your personal data; for information on exercising these rights or the Company's treatment of personal data, please email datarequests at jumptrading.com.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180823/9056645d/attachment-0002.htm>