[gpfsug-discuss] Spectrum Scale 5 and Reading Compressed Data

[Alec]

When comparing compression performance I see the following performance, is
anyone else getting significantly higher on any other systems?

Read Speeds:
lz4 with null fill data, ~ 90MB/s
lz4 with a SAS data set, ~40-50MB/s
z with null fill data, ~ 15MB/s
z with a SAS data set, ~ 5MB/s

While on a 4G page pool I tested each of these file sizes and got roughly
identical performance in all cases:
1 GB, 5 GB, and 10GB.

This was on an S824 (p8) with read performance typically going to 1.2GB/s
of read on a single thread (non-compressed).

Doing a "very limited test" in Production hardware E850, 8gb Page Pool,
with ~2.4 GB/s of read on a single thread (non-compressed) I got very
similar results.

In all cases the work was done from the NSD master, and due to the file
sizes and the difference in page pool, i'd expect the 1gb files to move at
a significantly faster pace if pagepool was a factor.

If anyone could tell me what performance they get on their platform and
what OS or Hardware they're using, I'd very much be interested.

I'm debating if using GPFS to migrate the files to a .gz compressed
version, and then providing a fifo mechanism to pipe through the compressed
data wouldn't be a better solution.

Alec

On Wed, Jan 20, 2021 at 2:10 PM Alec <anacreo at gmail.com> wrote:

> I see a lot of references to the page pool. Our page pool is only 8 gb and
> our files can be very large into the terrabytes.
>
> I will try increasing the page pool in dev to 2x a test file and see if
> the problem resolves.
>
> Any documentation on the correlation here would be nice.
>
> I will see if I can get rights for the debug as well.
>
> Alec
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20210122/205c67f2/attachment-0002.htm>