<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi All,
<div class=""><br class="">
</div>
<div class="">Wow - my query got more responses than I expected and my sincere thanks to all who took the time to respond!</div>
<div class=""><br class="">
</div>
<div class="">At this point in time we do have two GPFS filesystems … one which is basically “/home” and some software installations and the other which is “/scratch” and “/data” (former backed up, latter not). Both of them have their metadata on SSDs set
up as RAID 1 mirrors and replication set to two. But at this point in time all of the SSDs are in a single storage array (albeit with dual redundant controllers) … so the storage array itself is my only SPOF.</div>
<div class=""><br class="">
</div>
<div class="">As part of the hardware purchase we are in the process of making we will be buying a 2nd storage array that can house 2.5” SSDs. Therefore, we will be splitting our SSDs between chassis and eliminating that last SPOF. Of course, this includes
the new SSDs we are getting for our new /home filesystem.</div>
<div class=""><br class="">
</div>
<div class="">Our plan right now is to buy 10 SSDs, which will allow us to test 3 configurations: </div>
<div class=""><br class="">
</div>
<div class="">1) two 4+1P RAID 5 LUNs split up into a total of 8 LV’s (with each of my 8 NSD servers as primary for one of those LV’s and the other 7 as backups) and GPFS metadata replication set to 2. </div>
<div class=""><br class="">
</div>
<div class="">2) four RAID 1 mirrors (which obviously leaves 2 SSDs unused) and GPFS metadata replication set to 2. This would mean that only 4 of my 8 NSD servers would be a primary.</div>
<div class=""><br class="">
</div>
<div class="">3) nine RAID 0 / bare drives with GPFS metadata replication set to 3 (which leaves 1 SSD unused). All 8 NSD servers primary for one SSD and 1 serving up two.</div>
<div class=""><br class="">
</div>
<div class="">The responses I received concerning RAID 5 and performance were not a surprise to me. The main advantage that option gives is the most usable storage space for the money (in fact, it gives us way more storage space than we currently need) … but
if it tanks performance, then that’s a deal breaker.</div>
<div class=""><br class="">
</div>
<div class="">Personally, I like the four RAID 1 mirrors config like we’ve been using for years, but it has the disadvantage of giving us the least usable storage space … that config would give us the minimum we need for right now, but doesn’t really allow
for much future growth.</div>
<div class=""><br class="">
</div>
<div class="">I have no experience with metadata replication of 3 (but had actually thought of that option, so feel good that others suggested it) so option 3 will be a brand new experience for us. It is the most optimal in terms of meeting current needs plus
allowing for future growth without giving us way more space than we are likely to need). I will be curious to see how long it takes GPFS to re-replicate the data when we simulate a drive failure as opposed to how long a RAID rebuild takes.</div>
<div class=""><br class="">
</div>
<div class="">I am a big believer in Murphy’s Law (Sunday I paid off a bill, Wednesday my refrigerator died!) … and also believe that the definition of a pessimist is “someone with experience” <grin> … so we will definitely not set GPFS metadata replication
to less than two, nor will we use non-Enterprise class SSDs for metadata … but I do still appreciate the suggestions.</div>
<div class=""><br class="">
</div>
<div class="">If there is interest, I will report back on our findings. If anyone has any additional thoughts or suggestions, I’d also appreciate hearing them. Again, thank you!</div>
<div class=""><br class="">
</div>
<div class="">Kevin</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div class="">—</div>
<div class="">Kevin Buterbaugh - Senior System Administrator</div>
<div class="">Vanderbilt University - Advanced Computing Center for Research and Education</div>
<div class=""><a href="mailto:Kevin.Buterbaugh@vanderbilt.edu" class="">Kevin.Buterbaugh@vanderbilt.edu</a> - (615)875-9633</div>
<div class=""><br class="">
</div>
</div>
</div>
</body>
</html>